What is a transformer in AI? A transformer is a type of neural network that reads a whole piece of text at once, weighs the links between words, and then predicts or produces the next part. It became the standard design behind most modern language models after a 2017 research paper introduced it.
Key Takeaways
- A transformer is a neural network built around self-attention, not step-by-step reading.
- Self-attention lets each token weigh the relevance of every other token.
- The original design had an encoder for reading input and a decoder for producing output.
- BERT uses the encoder only, while GPT models use the decoder only.
- Transformers power text, code, search, and image models, but limits on context, compute, and accuracy remain.
Table of Contents
The Short Answer: What Is a Transformer in AI
A transformer is a machine learning architecture that turns input into a rich internal representation and then uses that representation to generate or classify output. The design was introduced in the 2017 transformer paper by Ashish Vaswani and seven co-authors at Google, and presented at NeurIPS 2017. The paper proposed a network based solely on attention, with no recurrence and no convolutions.
The same architecture also powers agentic AI systems and chatbots. A transformer does not store facts in a database or follow hand-written grammar rules. It learns patterns from large amounts of training text and applies those patterns to new input, for translation, summarisation, question answering, and code completion.
How Does Self-Attention Work?
Self-attention works by letting each token compare itself with every other token and assign a weight to each comparison. A token is a small piece of text, often a word or part of a word. In the sentence “The bank by the river flooded,” attention links “bank” more strongly to “river” than to a financial meaning. The weights are learned during training.
Attention was not new in 2017. Bahdanau and colleagues introduced an attention mechanism in 2014 as an addition to recurrent models for machine translation. The 2017 paper made a bolder move: it removed recurrence entirely and made attention the only mechanism for connecting tokens. That change allowed all tokens to be processed in parallel during training, rather than one step at a time.
What Is Multi-Head Attention?
Multi-head attention is self-attention run several times in parallel, with each “head” learning a different kind of relationship. One head might focus on grammar, such as the link between a subject and a verb. Another might focus on meaning, such as the link between a pronoun and the noun it refers to. The outputs are combined into a richer view of the same input.
What Are the Encoder and Decoder?
The encoder is the half of the original transformer that reads the input and builds a contextual representation of it. Each encoder layer applies self-attention and then a feed-forward network, a simple neural network applied to each position separately. The original paper stacked six identical encoder layers and six decoder layers. Positional encoding adds word-order information, because attention alone has no sense of sequence.
The decoder is the half that produces output one token at a time. It uses masked self-attention, so each position can attend only to tokens already produced, and cross-attention, so it can also look back at the encoder output. That structure suited the original task, machine translation: the encoder read a sentence in one language, and the decoder wrote the translation in another language.
Which Models Use Only One Half?
Many later models use only one half of the original design. BERT, introduced by Google in October 2018, uses only the encoder. It reads text in both directions and learns by predicting masked words, which makes it strong at understanding and classification. GPT models, introduced by OpenAI in 2018, use only the decoder. As of November 2026, most large language models, including the GPT family and Claude, follow the decoder-only pattern, according to published descriptions of those systems.
How Does a Transformer Process Text Step by Step?
A transformer processes text in a fixed sequence of stages, from raw input to finished output. The stages below describe text generation in a decoder-only model, the most common form in use as of November 2026.
- Split the input into tokens, such as whole words or word pieces, and turn each token into a vector called an embedding.
- Add positional encoding to each embedding, so the model keeps track of word order.
- Pass the embeddings through stacked layers of multi-head self-attention, so each token gathers context from the others.
- Pass the result through a feed-forward network at each position, with residual connections keeping the signal stable.
- Repeat the attention and feed-forward stages through many layers, building a deeper representation at each step.
- Use the final representation to predict the next token, add it to the sequence, and repeat until the output is complete.
Training uses the same layers in a different way. During training, the model sees large amounts of text and adjusts its internal weights to reduce prediction errors. During use, the weights stay fixed. Extra techniques, such as retrieval-augmented generation, can supply outside documents at run time, but the transformer remains the core engine that reads and writes the text.
Why Did Transformers Replace RNNs?
Transformers replaced recurrent neural networks, known as RNNs, because the older design processed text one step at a time and struggled with long-range links. An RNN passed a hidden state from one word to the next, so early words could fade by the end of a long sentence. Training was also slow, because each step waited for the previous step. The transformer removed that bottleneck by letting every token attend directly to every other token.
| Feature | Transformer | RNN | CNN |
|---|---|---|---|
| How input is read | All tokens at once | One token at a time | Small local windows |
| Long-range links | Direct, through self-attention | Indirect, through a fading hidden state | Indirect, built up over layers |
| Training on long text | Faster, in parallel | Slower, in sequence | Fast, but local in scope |
| Best known use | Language models and chatbots | Early sequence models | Image recognition |
| Main weakness | Compute cost grows with input length | Weak memory for distant context | Weak grasp of distant relationships |
The table shows a trade-off, not a total victory. Convolutional neural networks, known as CNNs, remain useful for local patterns in images. The transformer won the central role in language work because parallel training and direct long-range links scaled well as models and datasets grew.
Where Are Transformers Used Today?
Transformers are used in most modern language systems, from chatbots and search to translation, summarisation, and code assistants. GPT-3, released by OpenAI in 2020 with 175 billion parameters, showed that a large decoder-only transformer could handle many tasks from a few examples in the prompt. Later models in the GPT family, Claude, Gemini, and Llama all build on the same transformer foundation. The question how AI agents work also leads back to this design, because agents use language models to plan, call tools, and draft replies.
Transformers are also used beyond text. The Vision Transformer, introduced by Dosovitskiy and colleagues in 2020, splits an image into small patches and treats each patch like a token. That approach showed that the same architecture could work for image classification without convolutions, when trained on enough data. A related question, what is multimodel ai, points to systems that handle more than one kind of input, such as text plus images. The guide to multimodal generative AI explains how those combined systems work.
The broader picture is larger than a single model. The question what is an ai system describes a combination of models, data, software, and interfaces working together, and a transformer is often only one part of that combination. A search engine or voice assistant may use a transformer for language, plus databases, rules, and safety filters around it.
What Are the Limits of Transformers?
The limits fall into three areas: context length, compute cost, and factual reliability. The context window is the amount of input a model can consider at once. It is finite, so very long documents must be split, summarised, or retrieved in parts. Different models set it at different sizes as of November 2026.
Compute cost is the second limit. Self-attention compares each token with other tokens, so longer inputs need much more computation and memory. Training a large transformer also needs large datasets and specialised hardware. Smaller models reduce the cost, but the basic trade-off remains: more context and more capacity cost more compute.
Factual reliability is the third limit. A transformer predicts likely text, not verified truth, so it can produce fluent statements that are wrong. Those errors are often called hallucinations. The question what is the main goal of generative ai has a useful answer here: the goal is to produce new, useful content, such as text, images, or code, rather than to guarantee that every output is true. Retrieval, fact-checking, and human review help close that gap, but none removes it entirely.
Key Terms at a Glance
- Token: A small piece of text, such as a word or word part, that a model reads as one unit.
- Self-attention: A mechanism that lets each token weigh the relevance of every other token in the same input.
- Encoder: The part of a transformer that reads input and builds a contextual representation of it.
- Decoder: The part of a transformer that produces output one token at a time, using earlier tokens and encoder output.
- Context window: The maximum amount of input a model can consider in a single pass.
Conclusion
A transformer is a neural network that reads input in parallel, uses self-attention to weigh the links between tokens, and then generates or classifies output from the resulting representation. The design arrived in 2017, replaced recurrent models as the standard for language work, and now supports chatbots, search, translation, code tools, and image models. Its limits on context, compute, and factual accuracy remain real, so the best results come from pairing the architecture with retrieval, checks, and human judgement.
Frequently Asked Questions
How does self-attention work?
Self-attention works by scoring the relevance of every token to every other token and blending those scores into a new representation for each token. The scores are learned during training.
Is a transformer the same as a large language model?
No, a transformer is not the same as a large language model. A transformer is an architecture, while a large language model is a trained system built on that architecture, with specific data, size, and training methods.
Why are transformers so good at language?
Transformers are good at language because self-attention connects distant words directly and training runs in parallel across the whole input. Those two properties help the model learn grammar, meaning, and long-range context.
Can transformers handle images as well as text?
Yes, transformers can handle images as well as text. The Vision Transformer splits an image into patches, treats each patch like a token, and applies the same attention design used for language.
What is the biggest weakness of a transformer?
The biggest weakness is the combination of a finite context window and high compute cost for long inputs. Factual errors are a separate weakness, because the model predicts likely text rather than verified truth.
Does a transformer understand language like a person?
No, a transformer does not understand language like a person. It learns statistical patterns in text and uses those patterns to predict and generate language, without the lived experience or world knowledge that shapes human understanding.
Related Reading on Future AI Wire
Sources
- Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin (2017). 2017 transformer paper, arXiv:1706.03762.
- Vaswani, Ashish, et al. (2017). NeurIPS 2017 proceedings entry. Advances in Neural Information Processing Systems 30.
- Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805.
- Dosovitskiy, Alexey, et al. (2020). An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale. arXiv:2010.11929.

The Future AI Wire Editorial Team covers artificial intelligence, AI agents, agentic AI, and emerging technologies. We publish clear, research-informed explanations and practical insights to help readers understand AI developments, real-world applications, and the technologies shaping the future.

