Topic · Mechanisms

Positional encoding

Everything on this site about positional encoding, in the order to read it: a page is listed after the pages it builds on. The figures below are the ones those pages use to explain it; each is live: move a slider and it recomputes.

In reading order

  1. Paper · read first: Query, Key, Value — Dot-Product Attention from Scratch
  2. Read · read first: Attention Before Transformers — Score, Squash, Multiply

The figures these pages use

Pulled from the pages above. Move a slider and the figure recomputes, exactly as on its page.

From Attention Is All You Need — The Transformer, Block by Block › 2. The Encoder Layer

96
Where the parameters go. Parameters of the whole model, in millions, split by role. The feed-forward layers hold two thirds of every block; the shared embedding is a third of the base model. The number of heads is not a slider on purpose: it does not change the count.

From Attention Is All You Need — The Transformer, Block by Block › 4. Embeddings and Positional Encoding

710
Clock hands at different speeds. Left: four sine dimensions of the encoding against position, with the chosen pos marked; the first hand turns once every 6.28 positions, later hands ever more slowly. Right: the dot product between the encoding at pos and the encoding k steps later, computed from the two rows (solid) and from Σ cos(ω_i k) (dashed). Move pos: the right curve does not move, because the dot product depends on k alone.

From Attention Meets Pixels — Windows, Axes and the First Fully Attentional Networks › 1. Why n² Kills Images

7256
What one attention layer costs on an image. Entries of the attention matrix per layer, on a log10 scale, as the image side grows. Global attention is S⁴; a k×k window is S²·k²; two axial passes are 2·S²·m. Slide k and the span m and watch the two cheap curves cross: they tie where k² = 2m.

From Attention Meets Pixels — Windows, Axes and the First Fully Attentional Networks › 4. Axes: Attention One Axis at a Time

4121
Which pixels does one query see?. Scheme 0 global · 1 local 7×7 · 2 one-dimensional block · 3 two-dimensional block · 4 axial. The darkest cell is the query; the left grid is one layer, the right grid two stacked layers. Move the query to the border and watch the local and block sets get clipped while the axial cross never loses its reach.

Tags that go with it

Self-attention 2Efficient attention 1Transformers 1