Attention Is All You Need — The Transformer, Block by Block

Recurrent encoder-decoders compute $h_t=f(h_{t-1},x_t)$, one step after another, and "this inherently sequential nature precludes parallelization within training examples" (§1 of [VASW17]). The convolutional alternatives of the time, ByteNet [KALC16] and ConvS2S [GEHR17], run in parallel but need a stack of $O(\log n)$ or $O(n/k)$ layers before two distant positions can talk. The paper wants both at once, constant path length and parallel training, and gets there with "the first transduction model relying entirely on self-attention" (§2).

The whole model in one sentence: a transformer is the attention of the last read, wrapped in a block that is repeated (attention, then a small per-token MLP, each with a skip connection and a LayerNorm), fed with embeddings that have a position signal added, and trained with a careful recipe. Four modules, and I take them in that order.

Brush up on Query, Key, Value — Dot-Product Attention from Scratch: everything called $\mathrm{M}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}\mathrm{H}\mathrm{e}\mathrm{a}\mathrm{d}(·)$ below is Eq. 6 of that read and I treat it as a black box. Attention Before Transformers — Score, Squash, Multiply has Bahdanau's cross-attention, which reappears in the decoder. Residual connections [HE15] I assume you know.

1. Three Things the Paper Wanted

§4 of the paper compares layer types on three counts: "total computational complexity per layer", "the amount of computation that can be parallelized", measured as the minimum number of sequential operations, and "the path length between long-range dependencies", because "the shorter these paths… the easier it is to learn long-range dependencies" [VASW17].

Table 1 of the paper scores the three layer types on those counts, and the last read already reproduced it (Query, Key, Value — Dot-Product Attention from Scratch, its Table 1): self-attention connects any two positions in O(1) sequential steps over an O(1) path, where a recurrent layer needs O(n) of both and a stack of convolutions O(log_k n) layers, at the per-layer price of $n^2·d$ that the last read paid in full. The rest of this post is what the paper built on top of that bet.

2. The Encoder Layer

2.1 What It Is

Each of the N = 6 encoder layers has two sub-layers: multi-head self-attention, then a position-wise feed-forward network. Around each sub-layer there is a residual connection followed by a LayerNorm (§3.1). Every sub-layer outputs $d_{model}=512$ numbers per token so that the adds line up. Written out, with $X\in {ℝ}^{n\times d_{model}}$ the n tokens coming in:

• Attention mixes information across tokens: $Q=K=V=X$, every token looks at every token.
• The FFN then works on each token alone.
• Each step adds its result to what came in rather than replacing it, and a LayerNorm keeps the running sum at a fixed scale. Dropout (0.1) is applied to each sub-layer's output before the add (§5.4).
• The shape in is the shape out, so six of these stack.

Think of it as a two-step meeting repeated six times: everyone talks to everyone, then each thinks alone, with minutes kept (the residual) and the volume reset between rounds (the LayerNorm).

2.2 The Position-Wise Feed-Forward Network

Eq. 2 of the paper is two linear maps with a ReLU between them:

• The same $W_1,W_2$ are applied to every token row, and different layers have different weights. The paper's own bridge for the CNN reader: this is "two convolutions with kernel size 1" (§3.3).
• The hidden width is 4× the model width: 512 → 2048 → 512.

2.3 Layer Normalisation

LayerNorm [BA16] normalises one token's vector over its d features:

• $\mu$ and ${\sigma }^2$ are scalars computed over the 512 features of a single token, not over the batch and not over other tokens.
• $g,b\in {ℝ}^d$ are a learned gain and bias: 2d = 1,024 parameters per LayerNorm.

NOTE: LayerNorm is not BatchNorm. There are no batch statistics and no running averages, so a sentence of any length, alone, normalises the same way at train and test time. Also note the order: the 2017 paper normalises after the add, LayerNorm(x + Sublayer(x)). An Image Is Worth 16×16 Words — ViT, Patch by Patch and most later models move the LayerNorm in front of the sub-layer; I draw the 2017 version here.

2.4 The Numbers

Numbers are always satisfying to justify something, so let's count one encoder layer at $d_{model}=512,d_{ff}=2048,h=8$:

• Attention. $W^Q,W^K,W^V,W^O$ are each 512 × 512 = 262,144, so 4 × 262,144 = 1,048,576, plus 4 × 512 = 2,048 biases.
• FFN. 512 × 2048 + 2048 × 512 = 2,097,152, plus 2,048 + 512 = 2,560 biases.
• Two LayerNorms. 2 × (512 + 512) = 2,048.
• Total: 3,152,384 ≈ 3.15 M per layer. The FFN is 67 % of it, attention 33 %, LayerNorm 0.06 %. Two thirds of a transformer block is the MLP nobody talks about.

The figure counts the whole model by role. Change $d_{model}$ and N and watch the FFN bars stay at two thirds of every block while the embedding bar only follows $d_{model}$:

d_model = 2^k (d_ff = 4 d_model)

9

layers N

6

At the defaults the readout is 63.1 M next to the paper's 65 M; Section 6.2 closes that gap. The number of heads is not a slider because, as the last read showed, it does not change the count.

3. The Decoder Layer

The decoder also has N = 6 layers, each with a third sub-layer in the middle. With $Y\in {ℝ}^{m\times d_{model}}$ the m target tokens so far and $Z\in {ℝ}^{n\times d_{model}}$ the encoder output:

• Masked self-attention (line 1). The mask $M$ is added to $Q{K}^{\top }/\sqrt{d_k}$ before the softmax (§3.2.3). Since $\exp (-\infty )=0$, the weight a query at position i puts on any key j > i is exactly zero and the remaining weights still sum to 1.
• Encoder-decoder attention (line 2). Queries from the decoder, keys and values from the encoder output $Z$: this is Bahdanau's cross-attention [BAHD14] in Q/K/V form, and the only place the two stacks meet (Fig. 1).
• FFN (line 3), as in the encoder.

The targets are also shifted right by one position before they enter the decoder, so that the prediction at position i is made from positions < i only. Together with the mask this makes the model auto-regressive: at inference it emits one token, appends it, and runs again.

NOTE: The mask acts on the scores, before the softmax, as −∞. Zeroing the weights after the softmax would break the sum-to-one and is not what the paper does.

The decoder layer costs more than the encoder's: a second attention block and a third LayerNorm bring it to 2 × 1,050,624 + 2,099,712 + 3,072 = 4,204,032 ≈ 4.2 M parameters.

4. Embeddings and Positional Encoding

4.1 Embeddings

§3.4: the input embedding, the output embedding and the pre-softmax linear layer share one matrix, and the embeddings are multiplied by $\sqrt{d_{model}}$. The vocabulary is ≈ 37,000 byte-pair tokens for English–German and 32,000 word-pieces for English–French. At 37,000 × 512 = 18.9 M, the shared embedding is a third of the base model.

4.2 Why Positions at All?

Nothing in $\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(Q{K}^{\top }/\sqrt{d_k})V$ knows where a token sits: permute the input rows and the output rows permute with them. The paper therefore adds a positional encoding to the embeddings "at the bottoms of the encoder and decoder stacks" (§3.5), made of "sine and cosine functions of different frequencies":

• $pos$ is the token index 0 … n−1, $i$ indexes a sine/cosine pair, and $PE\in {ℝ}^{n\times d_{model}}$ is simply added to the embedding matrix.
• The wavelengths form "a geometric progression from 2π to 10000·2π": pair 0 turns once every 6.28 positions, pair 255 (for d = 512) once every ≈ 60,000.

Each pair of dimensions is a clock hand turning at its own speed. One fast hand alone is ambiguous, like a seconds hand without the minutes, but reading all 256 hands at once pins down the position uniquely, the way hours, minutes and seconds do.

4.3 Why Sinusoids: A Shift Is a Rotation

The paper chose this form because it "hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k, $P{E}_{pos+k}$ can be represented as a linear function of $P{E}_{pos}$" [VASW17]. Let's see why. Take one pair with frequency $\omega ={\omega }_i$ and use the angle-addition formulas:

• Line 1: shifting by k rotates the hand of pair i by the fixed angle ${\omega }_{i}k$, whatever $pos$ was.
• Line 2: stack the 2×2 rotations into one block-diagonal $R_k$; it depends on k and not on pos. So a weight matrix can learn "look k steps back" once, for every position.
• Line 3: the dot product between two encodings depends only on their distance k. It is largest at k = 0, where it equals d/2, and falls off from there.

d_model = 2^k

7

pos

10

Move the pos slider and the right-hand curve stays put: for d = 512 the dot product is 256 at k = 0, 249.1 at k = 1, 231.7 at k = 2, 189.6 at k = 5, 173.8 at k = 10 and 143.9 at k = 32, identical for pos = 10 and pos = 32. The second thing the paper hoped for was extrapolation: sinusoids "may allow the model to extrapolate to sequence lengths longer than the ones encountered during training", which a learned table cannot do.

NOTE: Both hopes are hypotheses the paper states and does not test. What it does test is learned versus sinusoidal positions, Table 3 row (E): 25.7 vs 25.8 dev BLEU, "nearly identical". Also, the encoding is added, not concatenated, and the embedding was scaled by √d_model first so the two live at a comparable scale.

5. The Recipe

§5 of the paper is short and every line of it matters.

• Data. WMT'14 English–German, 4.5 M sentence pairs; batches of about 25,000 source and 25,000 target tokens.
• Optimiser. Adam with β₁ = 0.9, β₂ = 0.98, ε = 10⁻⁹ and a schedule that warms up linearly for 4,000 steps and then decays with the inverse square root of the step (Eq. 3 of the paper):

• The peak sits at step 4,000: ${512}^{-0.5}\times {4000}^{-0.5}=0.0442\times 0.0158=7.0\times {10}^{-4}$. By step 100,000 it has decayed to $1.4\times {10}^{-4}$.
• Regularisation. Dropout 0.1 on every sub-layer output and on the embedding + position sum. Label smoothing of 0.1 [SZEG15], which "hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU" (§5.4).
• Hardware and time. 8 NVIDIA P100 GPUs. The base model: 100,000 steps at 0.4 s each, 12 hours. The big model ($d_{model}$ = 1024, $d_{ff}$ = 4096, 16 heads, dropout 0.3): 300,000 steps at 1.0 s, 3.5 days.
• Inference. The last 5 checkpoints (base) or 20 (big) are averaged; beam search with beam 4 and length penalty 0.6.

6. The Numbers

6.1 Translation

Table. 1 newstest2014 test BLEU and training cost, from Table 2 of [VASW17]. The GNMT, ConvS2S and ensemble rows are quoted as the paper tabulates them.

• The big model beats every previous single model and ensemble on En–De by "more than 2.0 BLEU", and on En–Fr at "less than 1/4 the training cost" of the previous best single model (§6.1).
• The FLOPs can be reproduced from §5.2 and footnote 5. 8 GPUs × 9.5 TFLOPS × 12 h × 3,600 s/h = 3.28 × 10¹⁸ ≈ the 3.3 · 10¹⁸ in the table. Big: 8 × 9.5 × 10¹² × 3.5 × 86,400 = 2.30 × 10¹⁹ ✓. ConvS2S on En–Fr used 6.5× the big model's compute; GNMT+RL on En–De used 7× the base model's for 2.7 BLEU less.

NOTE: §6.1 of the paper says 41.0 for the big En–Fr model while Table 2 says 41.8; I cite the table. The paper's Table 2 is newstest2014 test; its Table 3, my Table 2 below, is newstest2013 dev without checkpoint averaging, so 25.8 and 27.3 describe the same base model on different sets. And "12 hours" is the base model: the 28.4 BLEU model took 3.5 days.

6.2 Ablations

Table. 2 Variations on the base model, newstest2013 dev, from Table 3 of [VASW17]; the head and d_k rows are in the last read.

• Counting the whole base model with Section 2.4: 6 × 3.15 M (encoder, 18.9 M) + 6 × 4.2 M (decoder, 25.2 M) + the shared embedding 37,000 × 512 = 18.9 M gives 63.1 M, against the paper's 65 M; the gap is the exact vocabulary size and rounding. The slope checks too: the paper's N = 2 / 4 / 8 rows go 36 / 50 / 80 M, about 7.4 M per extra encoder-decoder layer pair, which is 3.15 + 4.2. The big model counts to 214 M against the paper's 213 M.
• Bigger is better on every axis the paper tried, dropout matters (−1.2 BLEU without it), and label smoothing buys 0.5 BLEU while making the perplexity worse (4.92 against 4.67): the model "learns to be more unsure" and translates better for it.
• Parsing (Table 4). A 4-layer transformer on WSJ reaches 91.3 F1 with 40k sentences and 92.7 semi-supervised, "despite the lack of task-specific tuning".

7. What Is, and Is Not, an Inductive Bias Here

"Transformers have no inductive bias" is wrong even in 2017. The FFN shares its weights across positions like a 1×1 convolution; the residual structure, the fixed sinusoids and the causal mask are all choices baked in. What is absent is locality and translation equivariance in the attention itself: a token may look anywhere, at any distance, with the same cost. That precise statement is the one An Image Is Worth 16×16 Words — ViT, Patch by Patch will need when it asks how much data it takes to make up for it.

8. What Came Next

The paper's own conclusion asks for "local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video". The vision community did exactly that, starting from the 50,176² number of the last read: restrict which keys a pixel may see, factorise attention along rows and columns, and replace convolutions one at a time. That is Attention Meets Pixels — Windows, Axes and the First Fully Attentional Networks. The encoder-decoder of Fig. 1 also returns unchanged in DETR — Detection as Set Prediction, No Anchors and No NMS, with learned queries in place of target tokens.

9. References

1. [VASW17] Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin. Attention Is All You Need. NIPS 2017.
2. [BA16] Ba, Kiros, Hinton. Layer Normalization. arXiv 2016.
3. [HE15] He, Zhang, Ren, Sun. Deep Residual Learning for Image Recognition. CVPR 2016.
4. [GEHR17] Gehring, Auli, Grangier, Yarats, Dauphin. Convolutional Sequence to Sequence Learning. ICML 2017.
5. [KALC16] Kalchbrenner, Espeholt, Simonyan, van den Oord, Graves, Kavukcuoglu. Neural Machine Translation in Linear Time. arXiv 2016.
6. [SZEG15] Szegedy, Vanhoucke, Ioffe, Shlens, Wojna. Rethinking the Inception Architecture for Computer Vision. CVPR 2016.
7. [BAHD14] Bahdanau, Cho, Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015.