The Transformer
Attention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin
The paper in one sentence
Transformers model relationships between tokens directly with attention, making sequence training far more parallel than recurrent architectures.
What the paper does
The paper introduces the Transformer, an encoder-decoder architecture built around attention instead of recurrent or convolutional sequence layers. Multi-head attention lets the model examine different representation subspaces in parallel, while positional encodings preserve token order. The result was stronger machine-translation quality with substantially more parallelizable training.
Why it matters to engineers
Most modern language models inherit this paper’s core computational pattern. Understanding self-attention, masking, residual connections, and positional information is therefore foundational for reasoning about model quality, context length, serving cost, and failure modes.
Key claims from the paper
- A sequence model can dispense with recurrence and convolutions while retaining strong translation quality.
- Multi-head attention allows the model to attend to information from different representation subspaces and positions.
- The architecture reduces sequential computation during training, enabling substantially more parallelism.
Architecture, step by step
- 1
Input representation
Token embeddings are combined with positional encodings so the model can use sequence order.
- 2
Encoder
Repeated blocks apply multi-head self-attention, a position-wise feed-forward network, residual connections, and normalization.
- 3
Decoder
Masked self-attention prevents access to future tokens; cross-attention then reads encoder outputs.
- 4
Prediction
A linear projection and softmax produce the probability distribution for the next output token.
Production takeaways
- Batching is effective during training because tokens in a sequence do not pass through a recurrent chain.
- Attention cost grows quadratically with sequence length in the standard formulation, so long contexts require careful memory planning.
- Causal masking is a correctness requirement for autoregressive generation, not merely an optimization detail.
- Residual paths and normalization are central to stable deep stacks and should be treated as architectural components.
Limitations and caveats
- Standard self-attention has quadratic time and memory complexity in sequence length.
- The original evaluation centered on machine translation; later model capabilities should not be attributed to this paper alone.
- Autoregressive inference remains sequential even though training can be highly parallel.
Interview questions to test understanding
- Why is Transformer training more parallelizable than RNN training?
- What does multi-head attention provide that a single attention head may not?
- How do causal masks differ from padding masks?
- Where does the quadratic context-length cost come from?