Topic · Mechanisms

Transformers

Everything on this site about transformers, in the order to read it: a page is listed after the pages it builds on. The figures below are the ones those pages use to explain it; each is live: move a slider and it recomputes.

In reading order

  1. Read · read first: Text2SQL — Part 3: Baseline Models
  2. Read · read first: Attention Before Transformers — Score, Squash, Multiply
  3. Paper
  4. Paper · read first: Object Detection — A Quick Read, Attention Meets Pixels — Windows, Axes and the First Fully Attentional Networks, Understanding Focal Loss —A Quick Read
  5. Paper · read first: Object Detection — A Quick Read
  6. Paper
  7. Paper
  8. Paper

The figures these pages use

Pulled from the pages above. Move a slider and the figure recomputes, exactly as on its page.

From Query, Key, Value — Dot-Product Attention from Scratch › 2. One Token, Step by Step

71
Who attends to whom. Rows are queries, columns are keys, both in sentence order: the, cat, sat, on, the, mat, because, it. The marked row is the chosen query; darker is more weight. The weights are from toy embeddings and hand-picked projections, not a trained model, so read the pattern, not the digits.

From Query, Key, Value — Dot-Product Attention from Scratch › 4. Why Divide by √d_k?

106
Why divide by √d_k. Random queries and keys with unit-variance components. Without the scale the largest weight climbs to 1 as the vectors get wider and the entropy collapses: an arg-max before any learning. With the scale the softness stays put at every width. The marker reads off the values at one d_k.

From Query, Key, Value — Dot-Product Attention from Scratch › 5. Multi-Head Attention

2
One sentence, several heads. Each grid is one head's 8×8 weight picture for 'the cat sat on the mat because it' (rows = queries, columns = keys, sentence order). The leftmost grid is a single head using all 16 dimensions; the others split the same 16 columns of W^Q, W^K into h slices. In the toy, slice 1 matches identical words, slice 2 sends 'it' to 'cat', slice 3 sends verbs and prepositions to nouns, slice 4 is random.

From Attention Is All You Need — The Transformer, Block by Block › 2. The Encoder Layer

96
Where the parameters go. Parameters of the whole model, in millions, split by role. The feed-forward layers hold two thirds of every block; the shared embedding is a third of the base model. The number of heads is not a slider on purpose: it does not change the count.

From Attention Is All You Need — The Transformer, Block by Block › 4. Embeddings and Positional Encoding

710
Clock hands at different speeds. Left: four sine dimensions of the encoding against position, with the chosen pos marked; the first hand turns once every 6.28 positions, later hands ever more slowly. Right: the dot product between the encoding at pos and the encoding k steps later, computed from the two rows (solid) and from Σ cos(ω_i k) (dashed). Move pos: the right curve does not move, because the dot product depends on k alone.

From DETR — Detection as Set Prediction, No Anchors and No NMS › 3. Bipartite Matching

10.0
The cost matrix the matcher sees. Rows are the three objects, columns are 8 of the 100 slots; darker is cheaper, the outlined cells are the chosen assignment. Raise the class weight and the matcher starts trusting what a slot believes over where its box is; slide slot 4 across the image and watch it steal an object.

From DETR — Detection as Set Prediction, No Anchors and No NMS › 4. The Set Loss

0.01.0
ℓ1 and GIoU as the box slides away. The prediction keeps the ground truth's shape and slides along x. The dotted line is where the overlap reaches zero: plain IoU loss goes flat there, GIoU keeps rising because the empty part of the enclosing box keeps growing. Push dy or the scale and the overlap vanishes sooner.

Tags that go with it

Language models 4Natural language 4Object detection 2Self-attention 2Set prediction 2Positional encoding 1