Transformer architecture refers to the full structural blueprint—encoder blocks, decoder blocks, or both—built on Self-Attention and feed-forward sub-layers that was introduced in 2017 and now underpins almost every state-of-the-art language, vision, and multimodal model. The encoder processes an input sequence into rich contextual embeddings; the decoder autoregressively generates an output sequence, attending both to its own previous outputs and to the encoder's representations. Models like BERT use only the encoder, GPT-style LLMs use only the decoder, while translation models use both halves together.

Think of it like a two-room library: the encoder is a reading room where incoming text is deeply cross-referenced and annotated; the decoder is a writing room where a scribe drafts a reply, periodically stepping back into the reading room to check the original notes.

from transformers import T5ForConditionalGeneration, T5Tokenizer
model = T5ForConditionalGeneration.from_pretrained("t5-small")
tokenizer = T5Tokenizer.from_pretrained("t5-small")
input_ids = tokenizer("translate English to French: Hello, world!", return_tensors="pt").input_ids
outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))  # "Bonjour, le monde!"

T5's encoder-decoder layout is the Transformer architecture in its complete, symmetric form.

The Transformer architecture enables Attention to scale to very long sequences (limited by Context Window) and forms the foundation for Pretraining paradigms like masked language modelling and next-token prediction. Variants such as Mixture of Experts replace dense feed-forward layers with sparse gating to increase capacity without proportionally increasing compute.

You'll encounter this architecture label in model cards, research papers, and framework documentation any time you work with modern NLP, computer vision transformers, or Multimodal systems.