CLIP — Learning Pictures From the Words Next to Them

A supervised vision model learns a fixed list of classes from a crowd-labelled dataset; add a task and you collect labels again. Meanwhile the web is full of pictures with text next to them, already describing what is in the picture, for free. CLIP by Radford et al. [RADF21] is the recipe that makes that text the supervision: encode the image, encode the caption, and train both encoders so that the right pairs match and the wrong pairs do not. Classification then becomes a question you ask in English. The whole system is in one picture:

NOTE: CLIP generates nothing. Two encoders and a dot product; it ranks captions it is given and writes none. DALL·E used CLIP to re-rank its samples, which is a different model.

Brush up first:

• Both towers are the transformer of Attention Is All You Need — The Transformer, Block by Block; the text tower is a 12-layer one with causal masking, nothing new.
• The image tower that matters is ViT-L/14 from An Image Is Worth 16×16 Words — ViT, Patch by Patch; the ResNets are in the paper for comparison, not for the series.
• Softmax as a soft lookup, and temperature as its sharpness knob: Query, Key, Value — Dot-Product Attention from Scratch. Here the softmax runs over $N$ candidates and I only define what the temperature multiplies.
• The other way to learn without labels, from the image alone: Masked Autoencoders — Hide 75 %, Encode the Rest. CLIP learns from the image's neighbour text instead.

1. Why Not Just Predict the Caption?

The first thing the authors tried was the obvious one: a transformer that writes the caption of each image, VirTex-style. It was too slow. "A 63 million parameter transformer language model ... learns to recognize ImageNet classes three times slower than a much simpler baseline that predicts a bag-of-words encoding of the same text" (§2.2, Figure 2). Their diagnosis: web text is "a wide variety of descriptions, comments, and related text that co-occur with images", so predicting the exact words is the wrong task. Predicting which text as a whole is paired with which image is enough, and swapping the prediction for that contrastive objective gave "a further 4× efficiency improvement in the rate of zero-shot transfer to ImageNet".

• Caption generation → bag-of-words: 3× faster.
• Bag-of-words → contrastive matching: 4× faster again.
• About 12× in total for the same zero-shot ImageNet accuracy. The task got easier and the model got better; that is the trade CLIP makes.

The contrastive form itself is a simplification of ConVIRT [ZHAN20], which did the same for medical images and reports; ALIGN [JIA21] published the same idea in the same month with noisier data.

2. The Pipeline, Module by Module

Symbols: a batch of $N$ pairs $(I_i,T_i)$; $f$ is the image encoder and $g$ the text encoder; $W_i$ and $W_t$ are learned linear projections into a shared space of $d_e$ dimensions (768 for ViT-L/14, 512 for the B models, Table 20 of the paper); $t$ is a learned scalar and $\tau =e^{-t}$ the temperature.

2.1 The Data

WIT, 400 million (image, text) pairs collected from the internet, built from 500,000 search queries with up to 20,000 pairs per query so that no single query dominates (§2.2). No labels. The only augmentation used anywhere is a random square crop.

2.2 Two Encoders, One Linear Projection Each

• Text. A 63 M-parameter transformer: 12 layers, width 512, 8 heads, a 49,152-token BPE vocabulary, at most 76 tokens, masked (causal) self-attention. The activation at the [EOS] token is the text feature (§2.4). For ViT-L/14 the text tower grows to 12 layers of width 768 with 12 heads.
• Image. Either a modified ResNet (RN50 up to RN50x64) or a ViT: B/32, B/16, L/14. The ResNet details (attention pooling, anti-aliasing) are one line in the paper and one line here; this series is about transformers.
• Projection. Each tower's output goes through a linear projection to the shared space and is L2-normalised. "We instead use only a linear projection"; the non-linear heads of the contrastive literature made no difference in training efficiency.

Let's write the two unit vectors:

• Because both are unit vectors, the dot product ${𝐮}_i·{𝐯}_j$ is the cosine similarity, a number in $[-1,1]$.
• The text side is the cheap half: the base text tower is 63 M parameters, against the 307 M of a ViT-L (Table 1 of An Image Is Worth 16×16 Words — ViT, Patch by Patch).

2.3 The N × N Logits and the Temperature

Take every image against every caption in the batch and scale by the learned temperature. This is the line logits = np.dot(I_e, T_e.T) * np.exp(t) of the paper's Figure 3 pseudocode:

• $L_{ii}$ is the real pair; the other $N^2-N$ entries are wrong pairs, and they are the negatives. Nobody labels them: they are whatever else landed in the batch.
• $\tau$ is learned, not a fixed hyper-parameter. It is "initialized to the equivalent of 0.07 ... and clipped to prevent scaling the logits by more than 100 which we found necessary to prevent training instability" (§2.5). So $e^{t_0}=1/0.07=14.3$ at the start, and $e^t\le 100$, i.e. $\tau \ge 0.01$, always.

Why the clip: a cosine lives in $[-1,1]$, so the logits live in $[-14.3,14.3]$ at initialisation and in $[-100,100]$ at the clip. Suppose the right caption is at logit 100 and a wrong one at 0. The softmax probability of the wrong one is about $e^{-100}\approx 4\times {10}^{-44}$, and by the saturation argument of Query, Key, Value — Dot-Product Attention from Scratch §4 its gradient is the same size: nothing. Let the scale run past 100 and every wrong pair goes invisible to the gradient. The paper says only that the clip was "necessary to prevent training instability"; this arithmetic is my reading of why, not theirs.

2.4 The Symmetric Cross-Entropy

Now the loss, and it is two losses. Row $i$ of $L$ is image $i$'s opinion of all $N$ captions; it must pick column $i$. Let's devise it step by step:

• Inside the log is a softmax over row $i$, evaluated at the diagonal. It is an $N$-way multiple-choice: "which of these $N$ captions is yours?"
• The paper's pseudocode writes this as cross_entropy_loss(logits, labels, axis=0) with labels = np.arange(n). The label of image $i$ is simply $i$.

Column $j$ is caption $j$'s opinion of all $N$ images, and it must pick row $j$. Same game, other direction, and the two are averaged:

• Eq. 3 normalises across a row, Eq. 4 across a column; the pseudocode's axis=0 and axis=1 are these two directions.
• Chance level for either direction is $\ln N$. With the paper's batch of 32,768 that is $\ln 32,768=10.40$ nats; the loss starts near 10.4 and has to fall from there, against 32,767 wrong captions per image, per step.

Play with the matrix below. Lower $\tau$ and every row sharpens; add pair noise and the diagonal stops being the obvious maximum. Watch what a sharp temperature does to a noisy batch: the loss explodes, which is the clip of Section 2.3 seen from the other side.

log10 τ (0.07 at init, 0.01 at the clip)

-1.155

pair noise σ

0.3

NOTE: the matrix is the expensive part. At N = 32,768 it has 32,768² = 1,073,741,824 entries, exactly 2³⁰: 4.29 GB in fp32, 2.15 GB in fp16, for one matrix, before the softmax and before any gradient. The paper shards it: each of the 256 GPUs computes "only the subset of the pairwise similarities necessary for their local batch", so one GPU holds 128 rows × 32,768 ≈ 4.2 M entries (about 17 MB in fp32) and gathers only the embeddings, 32,768 × 768 × 2 B ≈ 50 MB. The full matrix never exists anywhere.

2.5 Batch and Schedule

• Batch 32,768; 32 epochs; Adam with cosine decay; mixed precision. 400 M × 32 = 12.8 billion image–text views.
• The largest ResNet, RN50x64, took 18 days on 592 V100s (≈ 255,700 GPU-hours). ViT-L/14 took 12 days on 256 V100s (≈ 73,700 GPU-hours), then one extra epoch at 336 px. That last model, ViT-L/14@336px, is the one called "CLIP" in every result below.

3. Zero-Shot Transfer by Prompting

Here is the part that made people sit up. After pre-training there is no classifier head and no class vocabulary, so how do you classify? You write the classes down. For $K$ classes, put each class name into a sentence, embed the sentences once with the text tower, and let the image pick the nearest one:

• $𝒫$ is a set of prompt templates such as "A photo of a {label}."; $p({\text{name}}_k)$ is the template filled with class $k$. With a single template the sum has one term.
• The averaging happens in embedding space (§3.1.4), so the $K$ averaged vectors are computed once and the test-time cost is one dot product per class, whatever $\left|𝒫\right|$ is.

Why a sentence and not the bare word? Two reasons the paper gives:

• Polysemy. A lone "boxer" is a dog or an athlete; a lone "crane" is a bird or a machine. Context in the sentence picks one.
• Distribution gap. The text tower saw full sentences during pre-training, rarely a single word. "A photo of a {label}." puts the label back into the kind of text the model has seen, and alone it is worth +1.3 % on ImageNet. Ensembling 80 hand-written templates in embedding space adds +3.5 % more; together "almost 5 %" (Figure 4).

The figure below is a toy of that mechanism with made-up vectors; the only real numbers in it are the paper's gains quoted in the third tile. Step through the prompt set and watch the probability of the true class.

prompt set (1 = bare label, 2 = a photo of…, 3–5 = ensemble)

1

per-sentence wobble

0.3

NOTE: prompt engineering is part of the method, and it is worth five points. A "CLIP zero-shot" number quoted without the 80-template ensemble is a different system; compare like with like.

4. Results

4.1 ImageNet Without ImageNet

Zero-shot ImageNet top-1 for ViT-L/14@336px is 76.2 % (Table 11), with 95 % top-5. That matches the original ResNet-50, which was trained on the 1.28 million labelled ImageNet images; CLIP saw none of them. The previous best zero-shot result, Visual N-Grams, was 11.5 %.

Few-shot puts the same fact another way (§3.1.5, Figure 6): "zero-shot CLIP matches the average performance of a 4-shot linear classifier" trained on CLIP's own features across the suite, and on ImageNet it "matches the performance of a 16-shot linear classifier trained on the same feature space". Writing the class name down is worth sixteen labelled examples per class.

4.2 Twenty-Seven Datasets

The headline comparison is zero-shot CLIP against a supervised linear probe on ResNet-50 features, across 27 datasets (Figure 5). CLIP wins 16 of 27. The deltas, in percentage points:

Table. 1 Zero-shot CLIP minus a linear probe on ResNet-50 features, percentage points (Figure 5 of [RADF21]). Positive is a CLIP win.

• The wins are the datasets the web talks about: cars, food, countries, scenes, actions.
• The losses are the datasets the web does not: satellite crops (EuroSAT), distance to the car in front (KITTI), counting (CLEVR), traffic signs, lymph-node slides. Nobody captions a photo "satellite image of annual crop land". Zero-shot is not uniformly good; it is good where the text exists.

NOTE: "zero-shot" means no training examples of the target dataset, not no data. The 27-dataset suite was used repeatedly during development; the authors call it "undeniably co-adapted" (§6). And the paper's own estimate of the distance left: "around a 1000× increase in compute is required for zero-shot CLIP to reach overall state-of-the-art performance".

4.3 Zero-Shot Against a Linear Probe on CLIP Itself

A linear probe on CLIP's own frozen features (Table 10) is the ceiling that zero-shot is reaching for, and the gap tells you how much of a dataset is "in the text":

Table. 2 ViT-L/14@336px, zero-shot (Table 11 of [RADF21]) against a linear probe on the same features (Table 10 of [RADF21]), top-1 %.

• On pets and food the gap is two points: the text already says it all. On EuroSAT the same features reach 98.1 % with a probe but 59.6 % zero-shot: the features know, the prompt cannot ask.
• As a feature extractor CLIP also wins outright: the best CLIP beats Noisy-Student EfficientNet-L2 by +2.6 % on the 12-dataset suite and by about 5 % on all 27, winning 21 of 27 (§3.2).
• Scaling behaves (Figure 9): average zero-shot error falls log-linearly across a 44× range of compute over five CLIP models, the data-scale lesson of An Image Is Worth 16×16 Words — ViT, Patch by Patch again.

4.4 Robustness to Distribution Shift

The result I find most convincing is Figure 13. Take zero-shot CLIP and a ResNet-101 with the same 76.2 % ImageNet accuracy and move both to shifted test sets:

Table. 3 Natural distribution shift at equal ImageNet accuracy, top-1 % (Figure 13 of [RADF21]). "Zero-shot CLIP models shrink the gap by up to 75 %."

• ImageNet-A: 2.7 % → 77.1 %. A model that never fitted ImageNet's particular photographs has nothing to overfit to.

5. What Came Next

The symmetric softmax needs the global $N\times N$ normalisation, so SigLIP [ZHAI23] replaced it with a per-pair sigmoid that works at smaller batches and needs no all-gather of the matrix; LiT [ZHAI21] locked the image tower and trained only the text side; OpenCLIP [CHER22] and MetaCLIP [XU23] rebuilt the data in the open. Where the towers went from there, into Segment Anything, SigLIP 2 and the Perception Encoder, is the last post of the series, Vision Transformers Today — May 2022 to October 2026, Year by Year, which picks up in May 2022 with the pieces of this post and Masked Autoencoders — Hide 75 %, Encode the Rest on the table.

6. References

1. [RADF21] Radford et al. Learning Transferable Visual Models From Natural Language Supervision. 2021.
2. [ZHAN20] Zhang, Jiang, Miura, Manning, Langlotz. Contrastive Learning of Medical Visual Representations from Paired Images and Text. 2020.
3. [JIA21] Jia et al. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. 2021.
4. [ZHAI23] Zhai, Mustafa, Kolesnikov, Beyer. Sigmoid Loss for Language Image Pre-Training. 2023.
5. [ZHAI21] Zhai et al. LiT: Zero-Shot Transfer with Locked-image text Tuning. 2021.
6. [CHER22] Cherti et al. Reproducible scaling laws for contrastive language-image learning. 2022.
7. [XU23] Xu et al. Demystifying CLIP Data. 2023.
8. [DOSO20] Dosovitskiy et al. An Image is Worth 16×16 Words. 2020.
9. [VASW17] Vaswani et al. Attention Is All You Need. 2017.