Transformers in Vision

After this series you will be able to read any vision paper, count the FLOPs of any attention layer, write the losses of DETR and CLIP from memory, and explain why convolutions and transformers win or lose.

12 parts, in reading order · 24 interactive figures

Start with part 1

  1. Attention Before Transformers — Score, Squash, Multiply

    Part 1 · Read · Oct 2026 · 2 interactive figures

    • Bahdanau attention scores each source word against the decoder state, softmaxes the scores, and averages the encoder annotations into a context vector.
    • Squeeze-and-Excitation globally pools each channel, runs a two-layer MLP with reduction ratio 16, and applies independent sigmoid gates per channel.
    • The non-local block scores every position pair with embedded Gaussian similarity, softmaxes over all positions, and aggregates features into a global context map.
    • Bahdanau RNNsearch-50 reaches 26.75 BLEU on WMT'14 En→Fr, while the fixed-vector RNNencdec-50 scores 17.82.

  2. Query, Key, Value — Dot-Product Attention from Scratch

    Part 2 · Read · Oct 2026 · 3 interactive figures

    • Q, K, V all come from the same input X through three different learned projections W^Q, W^K, W^V.
    • W^Q and W^K must differ so a token can look for something without that thing looking back.
    • Scaling scores by 1/√d_k keeps softmax gradients healthy at initialization regardless of key width.
    • Eight heads run the same attention formula in parallel on 64-d slices, then concatenate outputs.

  3. Attention Is All You Need — The Transformer, Block by Block

    Part 3 · Paper · Oct 2026 · 2 interactive figures

    • Encoder layer: self-attention then per-token FFN, each with residual add and post-LayerNorm, repeated six times.
    • FFN is two linear maps (512→2048→512) with ReLU; same weights across tokens, different per layer.
    • LayerNorm normalizes each token's 512 features separately (no batch stats), with learned gain and bias.
    • Decoder layer adds masked self-attention and encoder-decoder cross-attention, totaling 4.2 M params.

  4. Attention Meets Pixels — Windows, Axes and the First Fully Attentional Networks

    Part 4 · Read · Oct 2026 · 2 interactive figures

    • Global attention on 224×224 pixels needs 2.5 billion matrix entries per layer, about 80 GB for 8 heads in fp32.
    • Image Transformer cuts cost to linear by restricting each query to a fixed memory block of preceding pixels in raster order.
    • Stand-alone attention replaces 3×3 convolutions with k×k windows and adds a learned vector per row and column offset.
    • Axial attention splits the 2D neighbourhood into two 1D passes, reducing 224×224 cost 112× versus global attention.

  5. DETR — Detection as Set Prediction, No Anchors and No NMS

    Part 5 · Paper · Oct 2026 · 2 interactive figures

    • DETR predicts 100 object queries in parallel, matched one-to-one to ground truth by Hungarian algorithm, removing anchors and NMS.
    • Matching cost combines class probability and box distance; class weight shifts assignment from geometry to confidence.
    • Box loss mixes L1 (weight 5) and generalized IoU (weight 2) to handle scale variance; no gradient flows through the discrete matcher.

  6. An Image Is Worth 16×16 Words — ViT, Patch by Patch

    Part 6 · Paper · Oct 2026 · 2 interactive figures

    • ViT changes only three things in the standard transformer encoder: patch input, learned 1-D positions, and a class token for read-out.
    • Patch embedding is a shared linear map (equivalent to P×P convolution with stride P) applied to flattened 16×16 patches.

  7. DeiT — Training a Vision Transformer on ImageNet Alone

    Part 7 · Paper · Oct 2026 · 2 interactive figures

    • Recipe swaps NLP defaults for strong augmentation: RandAugment, Mixup, CutMix, repeated augmentation, stochastic depth, lower weight decay.
    • Ablation shows removing Mixup and CutMix together costs 6.0 points; removing repeated augmentation costs 5.3 points; SGD instead of AdamW costs 7.3 points.

  8. Swin Transformer — Windows, Shifts and a Pyramid for Dense Prediction

    Part 8 · Paper · Oct 2026 · 2 interactive figures

    • 4×4 patches and 2×2 merging build a 4-stage pyramid at strides 4, 8, 16, 32.
    • Window attention with M=7 makes cost linear in tokens instead of quadratic.
    • Cyclic shift by 3 tokens every other layer connects windows without extra compute.
    • Relative position bias uses 169 learned values per head for 7×7 windows.

  9. Convolutions Strike Back — Was It the Attention or the Recipe?

    Part 9 · Read · Oct 2026 · 2 interactive figures

    • PVT keeps global attention but adds a pyramid and spatial-reduction attention (SRA) with ratios 8/4/2/1, cutting stage-1 cost 64× while staying quadratic overall.
    • CvT puts overlapping convolutions in the token embedding and depthwise-separable convolutions in Q/K/V projections, then drops all position embeddings.
    • CoAtNet merges convolution and attention into relative attention: a global scalar bias w_{i-j} added inside the softmax, giving translation equivariance plus input-adaptive weighting.

  10. Masked Autoencoders — Hide 75 %, Encode the Rest

    Part 10 · Paper · Oct 2026 · 2 interactive figures

    • MAE masks 75 % of patches, runs a ViT encoder on the remaining 25 %, and uses a tiny decoder to reconstruct pixels.
    • Encoder skips mask tokens entirely, cutting FLOPs 4.1× and boosting linear-probe accuracy from 59.6 to 73.5.
    • Decoder is only 8 layers of width 512 (9 % FLOPs per token of ViT-L) and is discarded after pre-training.
    • High mask ratio works because images are redundant; 15 % masking like BERT lets the model copy neighbours.

  11. CLIP — Learning Pictures From the Words Next to Them

    Part 11 · Paper · Oct 2026 · 2 interactive figures

    • CLIP trains on 400 million web image–caption pairs with a symmetric cross-entropy loss over an N × N similarity matrix, using a learned temperature clipped at 100× scale.
    • Contrastive matching is 12× more efficient than caption generation for the same zero-shot ImageNet accuracy, replacing word prediction with pair ranking.
    • ViT-L/14 image encoder and a 12-layer causal text transformer project to 768-d unit vectors; dot products become cosine similarities scaled by temperature τ = e⁻ᵗ.
    • Batch size 32,768 across 256 GPUs shards the 1B-entry logits matrix so each GPU holds only ~4.2M entries, avoiding a full 4.3 GB matrix in memory.

  12. Vision Transformers Today — May 2022 to October 2026, Year by Year

    Part 12 · Read · Oct 2026 · 1 interactive figure

    • FlashAttention tiles Q, K, V into SRAM and recomputes softmax online, cutting HBM traffic 40.3→4.4 GB on GPT-2 medium while keeping exact attention.
    • DiT replaces the U-Net with a plain ViT on latent patches and adaLN-Zero conditioning, reaching FID 2.27 on ImageNet 256² at 118.6 Gflops.
    • SAM combines an MAE ViT-H encoder with a promptable decoder emitting three masks, trained on the 1.1B-mask SA-1B dataset for foundation segmentation.

Other series