Topic · Efficiency

Efficient attention

Everything on this site about efficient attention, in the order to read it: a page is listed after the pages it builds on. The figures below are the ones those pages use to explain it; each is live: move a slider and it recomputes.

In reading order

  1. Read · read first: Attention Before Transformers — Score, Squash, Multiply, Attention Is All You Need — The Transformer, Block by Block
  2. Paper · read first: An Image Is Worth 16×16 Words — ViT, Patch by Patch, DeiT — Training a Vision Transformer on ImageNet Alone
  3. Read · read first: Masked Autoencoders — Hide 75 %, Encode the Rest, CLIP — Learning Pictures From the Words Next to Them
  4. Paper · read first: Object Detection — A Quick Read, Understanding Focal Loss —A Quick Read, DETR — Detection as Set Prediction, No Anchors and No NMS
  5. Paper

The figures these pages use

Pulled from the pages above. Move a slider and the figure recomputes, exactly as on its page.

From Attention Meets Pixels — Windows, Axes and the First Fully Attentional Networks › 1. Why n² Kills Images

7256
What one attention layer costs on an image. Entries of the attention matrix per layer, on a log10 scale, as the image side grows. Global attention is S⁴; a k×k window is S²·k²; two axial passes are 2·S²·m. Slide k and the span m and watch the two cheap curves cross: they tie where k² = 2m.

From Attention Meets Pixels — Windows, Axes and the First Fully Attentional Networks › 4. Axes: Attention One Axis at a Time

4121
Which pixels does one query see?. Scheme 0 global · 1 local 7×7 · 2 one-dimensional block · 3 two-dimensional block · 4 axial. The darkest cell is the query; the left grid is one layer, the right grid two stacked layers. Move the query to the border and watch the local and block sets get clipped while the axial cross never loses its reach.

From Swin Transformer — Windows, Shifts and a Pyramid for Dense Prediction › 2. Window Attention and the Cost of Looking Everywhere

2247
Global attention against window attention, stage 1. Multiply-adds of one attention layer in stage 1 of Swin-T (C = 96, 4 × 4-pixel tokens) as the input side grows, on a log scale. The dotted line is the Q, K, V and output projections, which both versions pay. At 224 px, Eq. 1 and Eq. 2 of the paper give 2.00 G against 0.145 G per layer.

From Swin Transformer — Windows, Shifts and a Pyramid for Dense Prediction › 3. Shifted Windows

82
Windows, shifted windows, and the cyclic shift. Left: the regular partition. Middle: the same map with the partition moved by s tokens, which leaves small windows all round the edges. Right: the map rolled up and left by s so the regular partition fits again; the darker strips are tokens that wrapped around, and the mask stops them from talking to their new neighbours. Table 4: the shift is worth +1.1 top-1 and +2.8 box AP; Table 5: cyclic 3.0 ms against padding 3.3 ms in stage 1.

From Vision Transformers Today — May 2022 to October 2026, Year by Year › 3. 2024: The State-Space Year

76816
Attention against a selective scan, as tokens grow. Left: multiply-adds per layer, log scale, from 64 to 65,536 tokens (224² at patch 16 is 196; 1024² is 4,096). Right: what each has to keep in memory. The dotted vertical line is N = 6D, MambaOut's definition of a long sequence: left of it the quadratic term is a minority of a block's cost.

From RT-DETR — DETRs Beat YOLOs, Encoder by Encoder

1 / 5 · v11 / 8 · whole pipeline
One pipeline, five versions. The RT-DETR pipeline for a 640 × 640 image, block by block. Step through the stages to see each block's output shape; switch the version to see which blocks it changes (rust outline), and switch to training to see the dashed parts that exist only while the model learns and are removed at inference.

From RT-DETR — DETRs Beat YOLOs, Encoder by Encoder › 1. What NMS Costs

1 / 6 · 0.0013 / 4 · 0.7
How many boxes NMS has to sort. Boxes that survive the confidence threshold on one image and go into NMS. The lower the threshold, the more boxes, the more time, and the better the AP. The last bar is RT-DETR, which has no NMS stage at all.

From RT-DETR — DETRs Beat YOLOs, Encoder by Encoder › 2. Why DETRs Were Slow

6406 / 6 · E
Where the encoder spends its attention. Counts: how many attention scores one encoder layer computes as the input grows, if every token of the three maps attended to every other, if each map attended only to itself, or if only the 20 × 20 map did (AIFI); the rust line is what a deformable encoder samples instead. These are counts, not time. Measured: the paper's own walk from variant A to E, AP against latency.

From RT-DETR — DETRs Beat YOLOs, Encoder by Encoder › 4. Query Selection: 300 Hands Up

0.80.752
Toy: picking the top K
060
A class score that knows how good its box is. The training target: for a matched position the loss is lowest when the class score p equals the IoU q of its box (left); for every other class it is pushed toward 0, gently near 0 (right). Picking the top K, a toy: made-up positions, picked by score; the more the score tracks the IoU, the more of the picks have good boxes.

From RT-DETR — DETRs Beat YOLOs, Encoder by Encoder › 5. The Decoder

(0.42, 0.5)0.30.362 / 3 · S4 · 40 × 400
Where one query reads. One query's box and the 32 points its 8 heads read on one map, where they start before training: four per head, from the centre out to the box's edge. Drag the centre, resize the box, switch the map: the points move with the box, and the shaded cells are the only ones this query reads. The layer slider refines the box toward a toy target with made-up offsets.

From RT-DETR — DETRs Beat YOLOs, Encoder by Encoder › 5. The Decoder

3 / 7 · R50, 6 layers6
Dropping decoder layers at inference. Train once with k decoder layers, then read the boxes after an earlier layer and skip the rest. Left: AP for the layer you stop at; right: latency. The scaled models are separate networks from the paper's appendix.

From RT-DETRv2 — A Bag of Freebies for the Decoder › 2. Points per Scale

(0.45, 0.4)
Point budget
4441 / 5 · S (R18)
Reading between cells. A toy 6 × 6 feature map. Drag the rust point: the bilinear read mixes the four cells around it with the weights shown, and the blue arrow is the way its slope pushes the point during training. Switch to the nearest-cell read: one cell, weight 1, and along the row the value becomes a staircase whose slope is 0, so the offsets get no signal. The point-budget sliders count the sampling points per image; Table 3 of the paper measured only equal counts, on the S model.

From RT-DETRv2 — A Bag of Freebies for the Decoder › 6. Results

5 / 5 · the RT-DETR line1 / 4 · v1
Which families to show
Speed against accuracy, one table at a time. Latency on a T4 GPU against COCO val2017 AP, as each paper reports it. Step through the tables: every view is one paper's own comparison, because the papers time the same models differently. Dotted verticals join two models that run the same network at the same cost.

Tags that go with it

Object detection 2Vision transformers 2Edge deployment 1Feature pyramids and necks 1Non-maximum suppression 1Positional encoding 1Segmentation 1Self-attention 1