Topic · Mechanisms

Set prediction

Everything on this site about set prediction, in the order to read it: a page is listed after the pages it builds on. The figures below are the ones those pages use to explain it; each is live: move a slider and it recomputes.

In reading order

  1. Paper · read first: Object Detection — A Quick Read, Attention Meets Pixels — Windows, Axes and the First Fully Attentional Networks, Understanding Focal Loss —A Quick Read, Attention Is All You Need — The Transformer, Block by Block
  2. Paper · read first: Object Detection — A Quick Read
  3. Paper · read first: Object Detection — A Quick Read, Understanding Focal Loss —A Quick Read
  4. Paper · read first: RT-DETRv2 — A Bag of Freebies for the Decoder

The figures these pages use

Pulled from the pages above. Move a slider and the figure recomputes, exactly as on its page.

From DETR — Detection as Set Prediction, No Anchors and No NMS › 3. Bipartite Matching

10.0
The cost matrix the matcher sees. Rows are the three objects, columns are 8 of the 100 slots; darker is cheaper, the outlined cells are the chosen assignment. Raise the class weight and the matcher starts trusting what a slot believes over where its box is; slide slot 4 across the image and watch it steal an object.

From DETR — Detection as Set Prediction, No Anchors and No NMS › 4. The Set Loss

0.01.0
ℓ1 and GIoU as the box slides away. The prediction keeps the ground truth's shape and slides along x. The dotted line is where the overlap reaches zero: plain IoU loss goes flat there, GIoU keeps rising because the empty part of the enclosing box keeps growing. Push dy or the scale and the overlap vanishes sooner.

From RT-DETR — DETRs Beat YOLOs, Encoder by Encoder

1 / 5 · v11 / 8 · whole pipeline
One pipeline, five versions. The RT-DETR pipeline for a 640 × 640 image, block by block. Step through the stages to see each block's output shape; switch the version to see which blocks it changes (rust outline), and switch to training to see the dashed parts that exist only while the model learns and are removed at inference.

From RT-DETR — DETRs Beat YOLOs, Encoder by Encoder › 1. What NMS Costs

1 / 6 · 0.0013 / 4 · 0.7
How many boxes NMS has to sort. Boxes that survive the confidence threshold on one image and go into NMS. The lower the threshold, the more boxes, the more time, and the better the AP. The last bar is RT-DETR, which has no NMS stage at all.

From RT-DETR — DETRs Beat YOLOs, Encoder by Encoder › 2. Why DETRs Were Slow

6406 / 6 · E
Where the encoder spends its attention. Counts: how many attention scores one encoder layer computes as the input grows, if every token of the three maps attended to every other, if each map attended only to itself, or if only the 20 × 20 map did (AIFI); the rust line is what a deformable encoder samples instead. These are counts, not time. Measured: the paper's own walk from variant A to E, AP against latency.

From RT-DETR — DETRs Beat YOLOs, Encoder by Encoder › 4. Query Selection: 300 Hands Up

0.80.752
Toy: picking the top K
060
A class score that knows how good its box is. The training target: for a matched position the loss is lowest when the class score p equals the IoU q of its box (left); for every other class it is pushed toward 0, gently near 0 (right). Picking the top K, a toy: made-up positions, picked by score; the more the score tracks the IoU, the more of the picks have good boxes.

From RT-DETR — DETRs Beat YOLOs, Encoder by Encoder › 5. The Decoder

(0.42, 0.5)0.30.362 / 3 · S4 · 40 × 400
Where one query reads. One query's box and the 32 points its 8 heads read on one map, where they start before training: four per head, from the centre out to the box's edge. Drag the centre, resize the box, switch the map: the points move with the box, and the shaded cells are the only ones this query reads. The layer slider refines the box toward a toy target with made-up offsets.

From RT-DETR — DETRs Beat YOLOs, Encoder by Encoder › 5. The Decoder

3 / 7 · R50, 6 layers6
Dropping decoder layers at inference. Train once with k decoder layers, then read the boxes after an earlier layer and skip the rest. Left: AP for the layer you stop at; right: latency. The scaled models are separate networks from the paper's appendix.

From RT-DETRv3 — More Positives Without Paying at Inference › 1. What One-to-One Matching Leaves on the Table

100
Positives per object, four ways. A toy picture with three objects (rust) and 40 queries (grey dots). Start from plain one-to-one matching, switch on query groups, the one-to-many set and the CNN head, and count how many predictions each object trains as positive. Toy data: the boxes, scores and the shaking of the extra groups are made up; the counting rules follow the paper and the released configuration.

From RT-DETRv3 — More Positives Without Paying at Inference › 3. Perturbed Query Groups

0.141
Who may look at whom. The self-attention mask of the decoder in training, shrunk to a few queries per group. Row = the query that looks, column = the query it may look at. Blue: allowed; grey: denoising queries; rust dot: a link removed at random in a perturbed group. Move the drop rate or draw again: only the perturbed groups change.

From RT-DETRv3 — More Positives Without Paying at Inference › 6. Results

What each part buys. The paper's own ablations as gains in AP over the baseline it names. Parts: one part at a time, then all three together. Groups: how many perturbed query groups. Schedules: v3 against v1 (and v2) per backbone and number of epochs.

From RT-DETRv3 — More Positives Without Paying at Inference › 6. Results

5 / 5 · the RT-DETR line1 / 4 · v1
Which families to show
Speed against accuracy, one table at a time. Latency on a T4 GPU against COCO val2017 AP, as each paper reports it. Step through the tables: every view is one paper's own comparison, because the papers time the same models differently. Dotted verticals join two models that run the same network at the same cost.

Tags that go with it

Object detection 4Transformers 2Efficient attention 1Non-maximum suppression 1Training recipes 1