Topic · Training

Pretraining at scale

Everything on this site about pretraining at scale, in the order to read it: a page is listed after the pages it builds on. The figures below are the ones those pages use to explain it; each is live: move a slider and it recomputes.

In reading order

  1. Paper · read first: Attention Meets Pixels — Windows, Axes and the First Fully Attentional Networks, Attention Is All You Need — The Transformer, Block by Block, DETR — Detection as Set Prediction, No Anchors and No NMS
  2. Paper · read first: Attention Is All You Need — The Transformer, Block by Block, Masked Autoencoders — Hide 75 %, Encode the Rest

The figures these pages use

Pulled from the pages above. Move a slider and the figure recomputes, exactly as on its page.

From An Image Is Worth 16×16 Words — ViT, Patch by Patch › 2. Patch Embedding, the Class Token and Positions

16224
An image as a sequence of patches. Each cell is one patch, i.e. one token. Shrink P and the sequence grows as 1/P²; the attention term grows as 1/P⁴ but stays a small share of the layer until P reaches 4. Raise the image side with P fixed, as ViT does at fine-tuning time, and N grows while the patch vector does not.

From An Image Is Worth 16×16 Words — ViT, Patch by Patch › 6. Data Scale Against Inductive Bias

6.12.9
Data first, then size. Left: Table 5 of the paper, ImageNet accuracy after pre-training on 1.3 M, 14 M and 300 M images; on ImageNet alone the Large model is below the Base one. Right: Table 6, the best accuracy reachable under a pre-training budget; the hybrid leads at small budgets, the two meet at 87.12 % near 1,600 exaFLOPs, and only ViT-H/14 goes past it. Move the two sliders and read the gap between the two ViT sizes and what a ResNet needs to match a ViT.

From CLIP — Learning Pictures From the Words Next to Them › 2. The Pipeline, Module by Module

-1.1550.3
One batch, N × N logits, the diagonal is the answer. Rows are images, columns are captions; each cell is the softmax probability the image gives that caption, darker is higher, the outlined cells are the true pairs. Lower τ sharpens every row; noisier pairs make the right answer harder to find.

From CLIP — Learning Pictures From the Words Next to Them › 3. Zero-Shot Transfer by Prompting

10.3
Dressing a label in a sentence (toy). Softmax over five classes for one toy image of a boxer. Template 1 is the bare word; 2 adds "a photo of a {label}."; 3 to 5 average more sentences in. Each single sentence lands a little off its concept (the wobble); averaging cancels that. Toy numbers; the paper's are +1.3 and +3.5 points on ImageNet.

Tags that go with it

Contrastive learning 1Image classification 1Vision and language 1Vision transformers 1