Topic · Training

Contrastive learning

Everything on this site about contrastive learning, in the order to read it: a page is listed after the pages it builds on. The figures below are the ones those pages use to explain it; each is live: move a slider and it recomputes.

In reading order

  1. Paper · read first: Attention Is All You Need — The Transformer, Block by Block, An Image Is Worth 16×16 Words — ViT, Patch by Patch, Masked Autoencoders — Hide 75 %, Encode the Rest
  2. Paper · read first: RT-DETRv4 — Borrowing a Foundation Model's Eyes

The figures these pages use

Pulled from the pages above. Move a slider and the figure recomputes, exactly as on its page.

From CLIP — Learning Pictures From the Words Next to Them › 2. The Pipeline, Module by Module

-1.1550.3
One batch, N × N logits, the diagonal is the answer. Rows are images, columns are captions; each cell is the softmax probability the image gives that caption, darker is higher, the outlined cells are the true pairs. Lower τ sharpens every row; noisier pairs make the right answer harder to find.

From CLIP — Learning Pictures From the Words Next to Them › 3. Zero-Shot Transfer by Prompting

10.3
Dressing a label in a sentence (toy). Softmax over five classes for one toy image of a boxer. Template 1 is the bare word; 2 adds "a photo of a {label}."; 3 to 5 average more sentences in. Each single sentence lands a little off its concept (the wobble); averaging cancels that. Toy numbers; the paper's are +1.3 and +3.5 points on ImageNet.

From RT-DETR-World — Naming Any Object in Real Time

1 / 5 · v11 / 8 · whole pipeline
One pipeline, five versions. The RT-DETR pipeline for a 640 × 640 image, block by block. Step through the stages to see each block's output shape; switch the version to see which blocks it changes (rust outline), and switch to training to see the dashed parts that exist only while the model learns and are removed at inference.

From RT-DETR-World — Naming Any Object in Real Time › 1. Open Vocabulary: the Classifier Becomes a List of Names

322
The classifier is a list of names. Toy vectors. Each rust arrow is one name the user typed, the blue arrow is one object query, and the list ranks the names by cosine. Turn the query; then add “wheelchair”: one new vector joins the list, no other number moves, and the top answer can change. “Wheelchair” sits between chair and wheel only for the picture.

From RT-DETR-World — Naming Any Object in Real Time › 6. Relation-Aware Negative Relaxation

0.250.80.07
Relaxing a related negative. One anchor from a toy batch: a red double-decker bus. The hollow bar is how hard the loss pulls toward its own description; the solid bars, how hard it pushes on each wrong one. Raise α and the city bus, which the teacher calls related, is pushed less; the spoon and the cat keep their full weight. At α = 0 this is the standard contrastive loss.

From RT-DETR-World — Naming Any Object in Real Time › 8. Results

1 / 6 · Fixed AP
Fixed AP, standard AP, speed. Solid bars are RT-DETR-World, hollow bars the real-time detectors it is compared with (YW = YOLO-World-v2.1, YE = YOLOE-v8, OD = OV-DEIM, W = RT-DETR-World). Switch between the two kinds of AP: the lead holds in Fixed AP, not in standard AP at every size. The last two views are the ablation of the supervision (Table 5).

Tags that go with it

Vision and language 2Knowledge distillation 1Object detection 1Pretraining at scale 1