Topic · Mechanisms

Vision transformers

Everything on this site about vision transformers, in the order to read it: a page is listed after the pages it builds on. The figures below are the ones those pages use to explain it; each is live: move a slider and it recomputes.

In reading order

  1. Paper · read first: Attention Meets Pixels — Windows, Axes and the First Fully Attentional Networks, Attention Is All You Need — The Transformer, Block by Block, DETR — Detection as Set Prediction, No Anchors and No NMS
  2. Paper
  3. Paper · read first: Attention Meets Pixels — Windows, Axes and the First Fully Attentional Networks
  4. Read
  5. Paper
  6. Read · read first: CLIP — Learning Pictures From the Words Next to Them

The figures these pages use

Pulled from the pages above. Move a slider and the figure recomputes, exactly as on its page.

From An Image Is Worth 16×16 Words — ViT, Patch by Patch › 2. Patch Embedding, the Class Token and Positions

16224
An image as a sequence of patches. Each cell is one patch, i.e. one token. Shrink P and the sequence grows as 1/P²; the attention term grows as 1/P⁴ but stays a small share of the layer until P reaches 4. Raise the image side with P fixed, as ViT does at fine-tuning time, and N grows while the patch vector does not.

From An Image Is Worth 16×16 Words — ViT, Patch by Patch › 6. Data Scale Against Inductive Bias

6.12.9
Data first, then size. Left: Table 5 of the paper, ImageNet accuracy after pre-training on 1.3 M, 14 M and 300 M images; on ImageNet alone the Large model is below the Base one. Right: Table 6, the best accuracy reachable under a pre-training budget; the hybrid leads at small budgets, the two meet at 87.12 % near 1,600 exaFLOPs, and only ViT-H/14 goes past it. Move the two sliders and read the gap between the two ViT sizes and what a ResNet needs to match a ViT.

From DeiT — Training a Vision Transformer on ImageNet Alone › 3. The Recipe

00
The DeiT recipe, one ingredient at a time. Points of top-1 lost when one ingredient is taken out of the DeiT-B recipe (Table 8 of the paper). RA = RandAugment, Mix = Mixup, Cut = CutMix, neither = no Mixup and no CutMix, Rep = repeated augmentation, RE = random erasing, SD = stochastic depth. Hollow bars are runs that did not train at all with the fixed hyper-parameters.

From DeiT — Training a Vision Transformer on ImageNet Alone › 4. Distillation: Soft and Hard

0.73.0
Soft and hard distillation as the teacher's confidence changes. Two classes, true label A, the student fixed at p_s. The soft term is a bowl with its bottom at p_t = p_s; the hard term is a step at p_t = 0.5 and flat everywhere else. Table 3 of the paper: soft distillation 81.8 %, hard distillation 83.0 %.

From Swin Transformer — Windows, Shifts and a Pyramid for Dense Prediction › 2. Window Attention and the Cost of Looking Everywhere

2247
Global attention against window attention, stage 1. Multiply-adds of one attention layer in stage 1 of Swin-T (C = 96, 4 × 4-pixel tokens) as the input side grows, on a log scale. The dotted line is the Q, K, V and output projections, which both versions pay. At 224 px, Eq. 1 and Eq. 2 of the paper give 2.00 G against 0.145 G per layer.

From Swin Transformer — Windows, Shifts and a Pyramid for Dense Prediction › 3. Shifted Windows

82
Windows, shifted windows, and the cyclic shift. Left: the regular partition. Middle: the same map with the partition moved by s tokens, which leaves small windows all round the edges. Right: the map rolled up and left by s so the regular partition fits again; the darker strips are tokens that wrapped around, and the mask stops them from talking to their new neighbours. Table 4: the shift is worth +1.1 top-1 and +2.8 box AP; Table 5: cyclic 3.0 ms against padding 3.3 ms in stage 1.

From Convolutions Strike Back — Was It the Attention or the Recipe? › 4. ConvNeXt: Take Swin's Design, Remove the Attention, Keep the Score

137
ResNet-50 to ConvNeXt-T, one step at a time. Top-1 on ImageNet-1k after each step of the ConvNeXt roadmap (Table 10 of the paper, three seeds), against Swin-T trained the same way. Step 1 touches only the training; everything after it is the network.

From Convolutions Strike Back — Was It the Attention or the Recipe? › 6. The Scoreboard

10
Four hybrids, three compute tiers. ImageNet-1k only, 224 px, 300 epochs, no distillation; each paper's own table (PVT Table 2, CvT Table 3, CoAtNet Table 4, ConvNeXt Table 1; references from DeiT Table 5, Swin Table 1, ConvNeXt Table 10). Top-1 is drawn as points above ResNet-50's original 76.1 %; R50 new is the same network with the DeiT/Swin recipe. Hollow bars are the reference models. Recipes and FLOP counters differ between papers, so read tiers, not decimals.

From Masked Autoencoders — Hide 75 %, Encode the Rest › 2. The Pipeline, Module by Module

0.751
Hide most of the picture, encode the rest. Three ways to pick which 14 × 14 patches the encoder sees (shaded) and which it never does (white). Random sampling is the paper's default; block and grid are the alternatives of its Table 1f. Slide the mask ratio and watch the encoder's bill fall with it.

From Masked Autoencoders — Hide 75 %, Encode the Rest › 3. Why 75 % and Not 15 %?

75
Accuracy against mask ratio (Figure 5). Left: fine-tuning top-1, with the from-scratch 82.5 % as the dashed line. Right: linear probing on the frozen encoder. The linear probe peaks at 75 % and spans almost 20 points; fine-tuning is flat from 50 to 75 %.

From Vision Transformers Today — May 2022 to October 2026, Year by Year › 3. 2024: The State-Space Year

76816
Attention against a selective scan, as tokens grow. Left: multiply-adds per layer, log scale, from 64 to 65,536 tokens (224² at patch 16 is 196; 1024² is 4,096). Right: what each has to keep in memory. The dotted vertical line is N = 6D, MambaOut's definition of a long sequence: left of it the quadratic term is a minority of a block's cost.

Tags that go with it

Efficient attention 2Self-supervised learning 2Training recipes 2Backbone design 1Convolutions 1Data augmentation 1Feature pyramids and necks 1Image classification 1