Topic · Training

Knowledge distillation

Everything on this site about knowledge distillation, in the order to read it: a page is listed after the pages it builds on. The figures below are the ones those pages use to explain it; each is live: move a slider and it recomputes.

In reading order

  1. Paper · read first: An Image Is Worth 16×16 Words — ViT, Patch by Patch
  2. Paper · read first: RT-DETRv3 — More Positives Without Paying at Inference
  3. Paper · read first: CLIP — Learning Pictures From the Words Next to Them

The figures these pages use

Pulled from the pages above. Move a slider and the figure recomputes, exactly as on its page.

From DeiT — Training a Vision Transformer on ImageNet Alone › 3. The Recipe

00
The DeiT recipe, one ingredient at a time. Points of top-1 lost when one ingredient is taken out of the DeiT-B recipe (Table 8 of the paper). RA = RandAugment, Mix = Mixup, Cut = CutMix, neither = no Mixup and no CutMix, Rep = repeated augmentation, RE = random erasing, SD = stochastic depth. Hollow bars are runs that did not train at all with the fixed hyper-parameters.

From DeiT — Training a Vision Transformer on ImageNet Alone › 4. Distillation: Soft and Hard

0.73.0
Soft and hard distillation as the teacher's confidence changes. Two classes, true label A, the student fixed at p_s. The soft term is a bowl with its bottom at p_t = p_s; the hard term is a step at p_t = 0.5 and flat everywhere else. Table 3 of the paper: soft distillation 81.8 %, hard distillation 83.0 %.

From RT-DETRv4 — Borrowing a Foundation Model's Eyes

1 / 5 · v11 / 8 · whole pipeline
One pipeline, five versions. The RT-DETR pipeline for a 640 × 640 image, block by block. Step through the stages to see each block's output shape; switch the version to see which blocks it changes (rust outline), and switch to training to see the dashed parts that exist only while the model learns and are removed at inference.

From RT-DETRv4 — Borrowing a Foundation Model's Eyes › 3. The Deep Semantic Injector

1
Matching the direction, not the length. Toy vectors in two dimensions; the real ones have 768 numbers at each of 400 cells. Drag the blue student's tip, then move the length slider: the cosine loss does not change at all, the squared error does. The arrows at the tip show which way each loss pushes (their sizes are in the tiles).

From RT-DETRv4 — Borrowing a Foundation Model's Eyes › 4. Gradient-guided Adaptive Modulation

8 / 11 · λ = 200
Fixed λ against the adaptive one. Each dot is one training run of DEIM-L with the alignment loss at a fixed weight λ (Tab. 7 of the paper; the λ axis is logarithmic). The two horizontal lines are the run without a teacher and the run with GAM. The shaded band is every λ that GAM used in the released RT-DETRv4-L training log; play the epochs to see where it was. Single runs: the paper reports no spread.

From RT-DETRv4 — Borrowing a Foundation Model's Eyes › 4. Gradient-guided Adaptive Modulation

0
for the rule by hand
1.53
Keeping AIFI's share of the gradient in a band. The released training log of RT-DETRv4-L, one point per epoch. The first chart is the share of the whole gradient that reaches AIFI, with the target band; the second is the weight λ on the alignment loss in force that epoch. Play the epochs and watch λ climb whenever the share falls below the band. Switch to "the rule by hand" to set the share yourself and compare the paper's rule with the released code's.

From RT-DETRv4 — Borrowing a Foundation Model's Eyes › 6. Results

5 / 5 · the RT-DETR line1 / 4 · v1
Which families to show
Speed against accuracy, one table at a time. Latency on a T4 GPU against COCO val2017 AP, as each paper reports it. Step through the tables: every view is one paper's own comparison, because the papers time the same models differently. Dotted verticals join two models that run the same network at the same cost.

From RT-DETR-World — Naming Any Object in Real Time › 1. Open Vocabulary: the Classifier Becomes a List of Names

322
The classifier is a list of names. Toy vectors. Each rust arrow is one name the user typed, the blue arrow is one object query, and the list ranks the names by cosine. Turn the query; then add “wheelchair”: one new vector joins the list, no other number moves, and the top answer can change. “Wheelchair” sits between chair and wheel only for the picture.

From RT-DETR-World — Naming Any Object in Real Time › 6. Relation-Aware Negative Relaxation

0.250.80.07
Relaxing a related negative. One anchor from a toy batch: a red double-decker bus. The hollow bar is how hard the loss pulls toward its own description; the solid bars, how hard it pushes on each wrong one. Raise α and the city bus, which the teacher calls related, is pushed less; the spoon and the cat keep their full weight. At α = 0 this is the standard contrastive loss.

From RT-DETR-World — Naming Any Object in Real Time › 8. Results

1 / 6 · Fixed AP
Fixed AP, standard AP, speed. Solid bars are RT-DETR-World, hollow bars the real-time detectors it is compared with (YW = YOLO-World-v2.1, YE = YOLOE-v8, OD = OV-DEIM, W = RT-DETR-World). Switch between the two kinds of AP: the lead holds in Fixed AP, not in standard AP at every size. The last two views are the ablation of the supervision (Table 5).

Tags that go with it

Object detection 2Training recipes 2Contrastive learning 1Data augmentation 1Vision and language 1Vision transformers 1