Topic · Mechanisms

Self-attention

Everything on this site about self-attention, in the order to read it: a page is listed after the pages it builds on. The figures below are the ones those pages use to explain it; each is live: move a slider and it recomputes.

In reading order

  1. Read · read first: Understanding Attention Modules: CBAM and BAM — A Quick Read
  2. Read
  3. Paper
  4. Read

The figures these pages use

Pulled from the pages above. Move a slider and the figure recomputes, exactly as on its page.

From Attention Before Transformers — Score, Squash, Multiply › 3. Squeeze-and-Excitation — Attention over Channels

28
A gate per channel, applied to the whole channel. Rows are the 8 channels, columns the 16 positions of a 4×4 map. The middle column is the gate vector s; the real block gets it from a learned 2-layer MLP, here it is a sigmoid of how far each channel's mean sits from the others. β = 0 gives every channel 0.5; large β drives the gates to 0 or 1.

From Attention Before Transformers — Score, Squash, Multiply › 4. Non-local Neural Networks — Pairwise Attention over Positions

50.15
One pixel against every other pixel. A 24×24 toy image: a bright blob on the left (col 5), a mid-grey blob in the middle (col 12) and a copy of the bright blob on the right (col 19). The marked cell is the query; every cell is shaded by the weight the query gives it. Slide the query through the left blob and watch its far-away twin light up. Put it on the grey blob and the bright blobs win instead: a plain dot product rewards brightness, not sameness, which is why the paper puts learned θ and φ in front of it.

From Query, Key, Value — Dot-Product Attention from Scratch › 2. One Token, Step by Step

71
Who attends to whom. Rows are queries, columns are keys, both in sentence order: the, cat, sat, on, the, mat, because, it. The marked row is the chosen query; darker is more weight. The weights are from toy embeddings and hand-picked projections, not a trained model, so read the pattern, not the digits.

From Query, Key, Value — Dot-Product Attention from Scratch › 4. Why Divide by √d_k?

106
Why divide by √d_k. Random queries and keys with unit-variance components. Without the scale the largest weight climbs to 1 as the vectors get wider and the entropy collapses: an arg-max before any learning. With the scale the softness stays put at every width. The marker reads off the values at one d_k.

From Query, Key, Value — Dot-Product Attention from Scratch › 5. Multi-Head Attention

2
One sentence, several heads. Each grid is one head's 8×8 weight picture for 'the cat sat on the mat because it' (rows = queries, columns = keys, sentence order). The leftmost grid is a single head using all 16 dimensions; the others split the same 16 columns of W^Q, W^K into h slices. In the toy, slice 1 matches identical words, slice 2 sends 'it' to 'cat', slice 3 sends verbs and prepositions to nouns, slice 4 is random.

From Attention Is All You Need — The Transformer, Block by Block › 2. The Encoder Layer

96
Where the parameters go. Parameters of the whole model, in millions, split by role. The feed-forward layers hold two thirds of every block; the shared embedding is a third of the base model. The number of heads is not a slider on purpose: it does not change the count.

From Attention Is All You Need — The Transformer, Block by Block › 4. Embeddings and Positional Encoding

710
Clock hands at different speeds. Left: four sine dimensions of the encoding against position, with the chosen pos marked; the first hand turns once every 6.28 positions, later hands ever more slowly. Right: the dot product between the encoding at pos and the encoding k steps later, computed from the two rows (solid) and from Σ cos(ω_i k) (dashed). Move pos: the right curve does not move, because the dot product depends on k alone.

From Attention Meets Pixels — Windows, Axes and the First Fully Attentional Networks › 1. Why n² Kills Images

7256
What one attention layer costs on an image. Entries of the attention matrix per layer, on a log10 scale, as the image side grows. Global attention is S⁴; a k×k window is S²·k²; two axial passes are 2·S²·m. Slide k and the span m and watch the two cheap curves cross: they tie where k² = 2m.

From Attention Meets Pixels — Windows, Axes and the First Fully Attentional Networks › 4. Axes: Attention One Axis at a Time

4121
Which pixels does one query see?. Scheme 0 global · 1 local 7×7 · 2 one-dimensional block · 3 two-dimensional block · 4 axial. The darkest cell is the query; the left grid is one layer, the right grid two stacked layers. Move the query to the border and watch the local and block sets get clipped while the axial cross never loses its reach.

Tags that go with it

Positional encoding 2Transformers 2Attention modules 1Efficient attention 1