Everything on this site about object detection, in the order to read it: a page is listed after the pages it builds on. The figures below are the ones those pages use to explain it; each is live: move a slider and it recomputes.
Cross-entropy for y = 1 and y = 0. Left: a foreground example. Right: a background example. Even a confident, correct prediction still carries some loss.
What each dense layer reads. Input channels seen by each layer of a dense block. The same early channels (and their gradients) are reused by every later layer, which is the duplication CSP removes.
Keep or suppress, by DIoU. The dashed box M is the one already kept. Slide the candidate B_i across it: ordinary NMS looks only at IoU, DIoU-NMS also asks how far apart the centres are.
α-balanced cross-entropy. α scales the foreground curve and 1 − α the background curve. It changes how much each class counts, not which examples are hard.
Focal loss for y = 1 and y = 0. Left: a foreground example. Right: a background example. A larger γ flattens the side where the model is already right.
The cost matrix the matcher sees. Rows are the three objects, columns are 8 of the 100 slots; darker is cheaper, the outlined cells are the chosen assignment. Raise the class weight and the matcher starts trusting what a slot believes over where its box is; slide slot 4 across the image and watch it steal an object.
ℓ1 and GIoU as the box slides away. The prediction keeps the ground truth's shape and slides along x. The dotted line is where the overlap reaches zero: plain IoU loss goes flat there, GIoU keeps rising because the empty part of the enclosing box keeps growing. Push dy or the scale and the overlap vanishes sooner.
Loss at one heatmap cell. Left: the centre cell wants Ŷ → 1. Right: a neighbouring cell wants Ŷ → 0, but the penalty shrinks as its Gaussian value Y approaches 1.
The rounding error the offset head recovers. Left: the fraction lost when a pixel coordinate is mapped to the low-resolution map. Right: the L1 loss on it is flat-sloped, so every error counts the same.
Weighting the three losses. Raw losses (left of each pair) and what each contributes to L_det after weighting. The paper uses λ_size = 0.1 and λ_off = 1.
Resolution at each pyramid level. Side of the feature map at each level for the chosen input. f_i mixes the coarse, semantic map from above with the fine map x_i beside it.
Cost of unrolling the pyramid T times. Backbone and FPN passes per image for each T. With shared weights the parameters stay the same; only compute grows.
One pipeline, five versions. The RT-DETR pipeline for a 640 × 640 image, block by block. Step through the stages to see each block's output shape; switch the version to see which blocks it changes (rust outline), and switch to training to see the dashed parts that exist only while the model learns and are removed at inference.
How many boxes NMS has to sort. Boxes that survive the confidence threshold on one image and go into NMS. The lower the threshold, the more boxes, the more time, and the better the AP. The last bar is RT-DETR, which has no NMS stage at all.
Where the encoder spends its attention. Counts: how many attention scores one encoder layer computes as the input grows, if every token of the three maps attended to every other, if each map attended only to itself, or if only the 20 × 20 map did (AIFI); the rust line is what a deformable encoder samples instead. These are counts, not time. Measured: the paper's own walk from variant A to E, AP against latency.
A class score that knows how good its box is. The training target: for a matched position the loss is lowest when the class score p equals the IoU q of its box (left); for every other class it is pushed toward 0, gently near 0 (right). Picking the top K, a toy: made-up positions, picked by score; the more the score tracks the IoU, the more of the picks have good boxes.
Where one query reads. One query's box and the 32 points its 8 heads read on one map, where they start before training: four per head, from the centre out to the box's edge. Drag the centre, resize the box, switch the map: the points move with the box, and the shaded cells are the only ones this query reads. The layer slider refines the box toward a toy target with made-up offsets.
Dropping decoder layers at inference. Train once with k decoder layers, then read the boxes after an earlier layer and skip the rest. Left: AP for the layer you stop at; right: latency. The scaled models are separate networks from the paper's appendix.
Reading between cells. A toy 6 × 6 feature map. Drag the rust point: the bilinear read mixes the four cells around it with the weights shown, and the blue arrow is the way its slope pushes the point during training. Switch to the nearest-cell read: one cell, weight 1, and along the row the value becomes a staircase whose slope is 0, so the offsets get no signal. The point-budget sliders count the sampling points per image; Table 3 of the paper measured only equal counts, on the S model.
5 / 5 · the RT-DETR line1 / 4 · v1Which families to show
Speed against accuracy, one table at a time. Latency on a T4 GPU against COCO val2017 AP, as each paper reports it. Step through the tables: every view is one paper's own comparison, because the papers time the same models differently. Dotted verticals join two models that run the same network at the same cost.
Positives per object, four ways. A toy picture with three objects (rust) and 40 queries (grey dots). Start from plain one-to-one matching, switch on query groups, the one-to-many set and the CNN head, and count how many predictions each object trains as positive. Toy data: the boxes, scores and the shaking of the extra groups are made up; the counting rules follow the paper and the released configuration.
Who may look at whom. The self-attention mask of the decoder in training, shrunk to a few queries per group. Row = the query that looks, column = the query it may look at. Blue: allowed; grey: denoising queries; rust dot: a link removed at random in a perturbed group. Move the drop rate or draw again: only the perturbed groups change.
What each part buys. The paper's own ablations as gains in AP over the baseline it names. Parts: one part at a time, then all three together. Groups: how many perturbed query groups. Schedules: v3 against v1 (and v2) per backbone and number of epochs.
Matching the direction, not the length. Toy vectors in two dimensions; the real ones have 768 numbers at each of 400 cells. Drag the blue student's tip, then move the length slider: the cosine loss does not change at all, the squared error does. The arrows at the tip show which way each loss pushes (their sizes are in the tiles).
Fixed λ against the adaptive one. Each dot is one training run of DEIM-L with the alignment loss at a fixed weight λ (Tab. 7 of the paper; the λ axis is logarithmic). The two horizontal lines are the run without a teacher and the run with GAM. The shaded band is every λ that GAM used in the released RT-DETRv4-L training log; play the epochs to see where it was. Single runs: the paper reports no spread.
Keeping AIFI's share of the gradient in a band. The released training log of RT-DETRv4-L, one point per epoch. The first chart is the share of the whole gradient that reaches AIFI, with the target band; the second is the weight λ on the alignment loss in force that epoch. Play the epochs and watch λ climb whenever the share falls below the band. Switch to "the rule by hand" to set the share yourself and compare the paper's rule with the released code's.
The classifier is a list of names. Toy vectors. Each rust arrow is one name the user typed, the blue arrow is one object query, and the list ranks the names by cosine. Turn the query; then add “wheelchair”: one new vector joins the list, no other number moves, and the top answer can change. “Wheelchair” sits between chair and wheel only for the picture.
Relaxing a related negative. One anchor from a toy batch: a red double-decker bus. The hollow bar is how hard the loss pulls toward its own description; the solid bars, how hard it pushes on each wrong one. Raise α and the city bus, which the teacher calls related, is pushed less; the spoon and the cat keep their full weight. At α = 0 this is the standard contrastive loss.
Fixed AP, standard AP, speed. Solid bars are RT-DETR-World, hollow bars the real-time detectors it is compared with (YW = YOLO-World-v2.1, YE = YOLOE-v8, OD = OV-DEIM, W = RT-DETR-World). Switch between the two kinds of AP: the lead holds in Fixed AP, not in standard AP at every size. The last two views are the ablation of the supervision (Table 5).