RT-DETR

You can explain how each RT-DETR version makes DETRs faster and stronger: v1 beats YOLOs with a hybrid encoder and query selection, v2 adds decoder dials and training freebies, v3 adds training-only positives, v4 distills DINOv3 features, and World enables open-vocabulary detection.

5 parts, in reading order · 17 interactive figures

Start with part 1

  1. RT-DETR — DETRs Beat YOLOs, Encoder by Encoder

    Part 1 · Paper · Oct 2026 · 6 interactive figures

    • AIFI runs one attention layer on the 20×20 map only, cutting encoder tokens from 8,400 to 400.
    • CCFF fuses three scales with a PANet-style convolution path and RepBlocks that merge to one kernel at inference.
    • Query selection picks the best 300 of 8,400 positions from the fused features instead of using learned queries.
    • Six deformable decoder layers refine 300 boxes end-to-end with no NMS; last layers can be dropped for speed.

  2. RT-DETRv2 — A Bag of Freebies for the Decoder

    Part 2 · Paper · Oct 2026 · 3 interactive figures

    • v2 keeps v1's network and changes four things: points per scale, optional nearest-cell read, augmentation stop, and backbone lr per size.
    • v1 reads 8 heads × 3 scales × 4 points = 96 points per query per layer; v2 replaces fixed 4 with per-scale counts n_l.
    • Nearest-cell read freezes offsets after extra fine-tune, drops 0.5–0.7 AP, and removes the bilinear grid_sample operator for deployment.
    • Augmentation stop and per-size backbone lr cost nothing at inference; fewer points costs 0.6 AP at quarter budget; nearest-cell read costs 0.5–0.7 AP.

  3. RT-DETRv3 — More Positives Without Paying at Inference

    Part 3 · Paper · Oct 2026 · 5 interactive figures

    • One-to-one matching gives each object only one positive query per decoder layer; v3 adds three training-only helpers that raise this to 8 query positives plus up to 13 encoder grid points per object.
    • A PP-YOLOE head on the encoder's three feature maps supplies dense one-to-many supervision via ATSS then task-aligned assignment (top 13 points), adding +0.9 AP on R18 before being discarded at inference.
    • Perturbed query groups copy the query set four times, each with a random self-attention mask that breaks ties differently so multiple near-duplicate queries can become positive across groups.
    • The paper's mask formula multiplies logits by 0/1 before softmax, but multiplying by 0 leaves e^0=1 in the softmax; the released code instead adds a large negative bias to truly drop links.

  4. RT-DETRv4 — Borrowing a Foundation Model's Eyes

    Part 4 · Paper · Oct 2026 · 5 interactive figures

    • AIFI receives only 0.5 % of the training gradient in a well-trained baseline, creating a semantic bottleneck where the layer every later stage depends on learns almost nothing from the loss.
    • RT-DETRv4 uses a frozen DINOv3 ViT-B/16 as a teacher for F5, through a linear projector and a cosine loss, then discards the teacher and projector before deployment.
    • The adaptive weight GAM retunes λ once per epoch so AIFI's gradient share stays in a target band, keeping the teacher's pull steady throughout training.
    • In the ablation, injecting at F5 alone is the only position that helps (53.8 → 54.3 AP); backbone injections do not; the deployed v4-S/M/L/X gains +0.5 to +1.0 AP over DEIM for 6–16 % extra training time.

  5. RT-DETR-World — Naming Any Object in Real Time

    Part 5 · Paper · Oct 2026 · 4 interactive figures

    • Open-vocabulary detection replaces 80 fixed COCO classes with text vectors from any names typed at test time.
    • Deploys frozen DINOv3 ViT backbone with 3-layer DETR decoder and MiniLMv2 text encoder (66.6M params).
    • An LLM2CLIP-Llama-3.1-8B teacher supplies precomputed embeddings for the alignment losses and the similarities that relax negatives, in training only.

Other series