Topic · Training

Training recipes

Everything on this site about training recipes, in the order to read it: a page is listed after the pages it builds on. The figures below are the ones those pages use to explain it; each is live: move a slider and it recomputes.

In reading order

  1. Paper · read first: YOLOv4 — Version 0: Introduction
  2. Paper · read first: An Image Is Worth 16×16 Words — ViT, Patch by Patch
  3. Read · read first: Swin Transformer — Windows, Shifts and a Pyramid for Dense Prediction
  4. Paper · read first: YOLOv4 — Version 3: Proposed Workflow
  5. Paper · read first: Object Detection — A Quick Read
  6. Paper · read first: Simple, Powerful, and Fast— RegNet Architecture from Facebook AI
  7. Paper · read first: RT-DETR — DETRs Beat YOLOs, Encoder by Encoder
  8. Paper
  9. Paper
  10. Project

The figures these pages use

Pulled from the pages above. Move a slider and the figure recomputes, exactly as on its page.

From DeiT — Training a Vision Transformer on ImageNet Alone › 3. The Recipe

00
The DeiT recipe, one ingredient at a time. Points of top-1 lost when one ingredient is taken out of the DeiT-B recipe (Table 8 of the paper). RA = RandAugment, Mix = Mixup, Cut = CutMix, neither = no Mixup and no CutMix, Rep = repeated augmentation, RE = random erasing, SD = stochastic depth. Hollow bars are runs that did not train at all with the fixed hyper-parameters.

From DeiT — Training a Vision Transformer on ImageNet Alone › 4. Distillation: Soft and Hard

0.73.0
Soft and hard distillation as the teacher's confidence changes. Two classes, true label A, the student fixed at p_s. The soft term is a bowl with its bottom at p_t = p_s; the hard term is a step at p_t = 0.5 and flat everywhere else. Table 3 of the paper: soft distillation 81.8 %, hard distillation 83.0 %.

From Convolutions Strike Back — Was It the Attention or the Recipe? › 4. ConvNeXt: Take Swin's Design, Remove the Attention, Keep the Score

137
ResNet-50 to ConvNeXt-T, one step at a time. Top-1 on ImageNet-1k after each step of the ConvNeXt roadmap (Table 10 of the paper, three seeds), against Swin-T trained the same way. Step 1 touches only the training; everything after it is the network.

From Convolutions Strike Back — Was It the Attention or the Recipe? › 6. The Scoreboard

10
Four hybrids, three compute tiers. ImageNet-1k only, 224 px, 300 epochs, no distillation; each paper's own table (PVT Table 2, CvT Table 3, CoAtNet Table 4, ConvNeXt Table 1; references from DeiT Table 5, Swin Table 1, ConvNeXt Table 10). Top-1 is drawn as points above ResNet-50's original 76.1 %; R50 new is the same network with the DeiT/Swin recipe. Hollow bars are the reference models. Recipes and FLOP counters differ between papers, so read tiers, not decimals.

From RT-DETRv2 — A Bag of Freebies for the Decoder

1 / 5 · v11 / 8 · whole pipeline
One pipeline, five versions. The RT-DETR pipeline for a 640 × 640 image, block by block. Step through the stages to see each block's output shape; switch the version to see which blocks it changes (rust outline), and switch to training to see the dashed parts that exist only while the model learns and are removed at inference.

From RT-DETRv2 — A Bag of Freebies for the Decoder › 2. Points per Scale

(0.45, 0.4)
Point budget
4441 / 5 · S (R18)
Reading between cells. A toy 6 × 6 feature map. Drag the rust point: the bilinear read mixes the four cells around it with the weights shown, and the blue arrow is the way its slope pushes the point during training. Switch to the nearest-cell read: one cell, weight 1, and along the row the value becomes a staircase whose slope is 0, so the offsets get no signal. The point-budget sliders count the sampling points per image; Table 3 of the paper measured only equal counts, on the S model.

From RT-DETRv2 — A Bag of Freebies for the Decoder › 6. Results

5 / 5 · the RT-DETR line1 / 4 · v1
Which families to show
Speed against accuracy, one table at a time. Latency on a T4 GPU against COCO val2017 AP, as each paper reports it. Step through the tables: every view is one paper's own comparison, because the papers time the same models differently. Dotted verticals join two models that run the same network at the same cost.

From RT-DETRv3 — More Positives Without Paying at Inference › 1. What One-to-One Matching Leaves on the Table

100
Positives per object, four ways. A toy picture with three objects (rust) and 40 queries (grey dots). Start from plain one-to-one matching, switch on query groups, the one-to-many set and the CNN head, and count how many predictions each object trains as positive. Toy data: the boxes, scores and the shaking of the extra groups are made up; the counting rules follow the paper and the released configuration.

From RT-DETRv3 — More Positives Without Paying at Inference › 3. Perturbed Query Groups

0.141
Who may look at whom. The self-attention mask of the decoder in training, shrunk to a few queries per group. Row = the query that looks, column = the query it may look at. Blue: allowed; grey: denoising queries; rust dot: a link removed at random in a perturbed group. Move the drop rate or draw again: only the perturbed groups change.

From RT-DETRv3 — More Positives Without Paying at Inference › 6. Results

What each part buys. The paper's own ablations as gains in AP over the baseline it names. Parts: one part at a time, then all three together. Groups: how many perturbed query groups. Schedules: v3 against v1 (and v2) per backbone and number of epochs.

From RT-DETRv4 — Borrowing a Foundation Model's Eyes › 3. The Deep Semantic Injector

1
Matching the direction, not the length. Toy vectors in two dimensions; the real ones have 768 numbers at each of 400 cells. Drag the blue student's tip, then move the length slider: the cosine loss does not change at all, the squared error does. The arrows at the tip show which way each loss pushes (their sizes are in the tiles).

From RT-DETRv4 — Borrowing a Foundation Model's Eyes › 4. Gradient-guided Adaptive Modulation

8 / 11 · λ = 200
Fixed λ against the adaptive one. Each dot is one training run of DEIM-L with the alignment loss at a fixed weight λ (Tab. 7 of the paper; the λ axis is logarithmic). The two horizontal lines are the run without a teacher and the run with GAM. The shaded band is every λ that GAM used in the released RT-DETRv4-L training log; play the epochs to see where it was. Single runs: the paper reports no spread.

From RT-DETRv4 — Borrowing a Foundation Model's Eyes › 4. Gradient-guided Adaptive Modulation

0
for the rule by hand
1.53
Keeping AIFI's share of the gradient in a band. The released training log of RT-DETRv4-L, one point per epoch. The first chart is the share of the whole gradient that reaches AIFI, with the target band; the second is the weight λ on the alignment loss in force that epoch. Play the epochs and watch λ climb whenever the share falls below the band. Switch to "the rule by hand" to set the share yourself and compare the paper's rule with the released code's.

Tags that go with it

Object detection 7Data augmentation 4Backbone design 2Knowledge distillation 2Vision transformers 2Convolutions 1Edge deployment 1Efficient attention 1