RT-DETRv3 — More Positives Without Paying at Inference

In RT-DETR every object teaches one query per decoder layer what it looks like. Every other query sitting on the same dog is told "nothing here". A YOLO head, on the same image, tells a dozen grid points "dog" at once. RT-DETRv3 [WANG24] (Wang, Xia, Lv and Shi at Baidu, WACV 2025) puts the diagnosis in one line: the Hungarian matching "provides sparse supervision during training, leading to insufficient training of both the encoder and decoder" (§1).

The fix is three helpers that exist only while the model learns:

• a CNN one-to-many head on the encoder's three maps, so the encoder gets dense supervision;
• perturbed query groups: copies of the query set, each with randomly masked self-attention, each matched one-to-one on its own;
• a one-to-many query set that shares the decoder and is matched to every object repeated four times.

At inference all three are gone, and what ships is v1's network: same parameters, same GFLOPs, same latency (Table 1 of [WANG24]). In the paper's words, "all aforementioned modules are training-only" (Abstract). Switch the figure below between training and inference: the dashed blocks are the whole of v3.

version◀3 / 5 · v3▶ Nextstage◀7 / 8 · decoder layers▶ Next▶modeinferencetrainingmodel sizeR18 / SR34 / MR50 / LR101 / Xwhole pipelinev1 at inferencev1 in trainingv2 at inferencev3 in trainingv3 at inferencev4 in trainingv4 at inferenceWorld in trainingWorld at inferenceReset

Brush up before you start ;)

• One-to-one (Hungarian) matching, the set loss and why DETRs need no NMS: DETR — Detection as Set Prediction, No Anchors and No NMS §3–4. I will not re-derive the matcher here.
• Query selection, the focal-style matching cost, the varifocal loss and denoising queries: RT-DETR — DETRs Beat YOLOs, Encoder by Encoder.
• What v2 changed (decoder sampling, the training recipe), so you know which baseline v3 is compared against: RT-DETRv2 — A Bag of Freebies for the Decoder.
• Dense heads, anchor points and label assignment: Object Detection — A Quick Read.

1. What One-to-One Matching Leaves on the Table

Picture a classroom with one question per pupil who raised a hand, and the rule that each question goes to exactly one hand. Five hands go up for the dog; one gets the question, four are told to put their hands down. Those four were not wrong, they were just second. That is a positive (a prediction trained to match an object) against a negative (trained to say "no object").

• One-to-one matching is what lets RT-DETR skip NMS: the decoder learns to put exactly one box on each object (DETR — Detection as Set Prediction, No Anchors and No NMS §5).
• The price is that each object produces one positive per decoder layer. The gradient that tells the network "this is what a dog looks like" flows through one query, and through the encoder only via that query's reads.
• The paper blames this for two things: an under-trained encoder and decoder (§1), and the slow convergence of RT-DETR and RT-DETRv2 (§2.2). It also notes that v2's gains came with a longer training time (§1).

The paper never counts the positives. Let's count them from the released R18 configuration [XIA24], one ground-truth box, at the last decoder layer:

• $N$: one-to-one query groups (the main one plus 3 perturbed in the R18 and R34 configs; plus 2 for R50); each gives exactly one positive per object.
• $m$: copies of each object in the one-to-many set (§3.5, default 4).
• $k$: grid points the CNN head keeps per object: 13 in the config's task-aligned assigner, which takes over at epoch 30. Before that the ATSS assigner keeps a varying number out of 27 candidates (9 per level).
• RT-DETR has $P_{\text{dec}}=1$ and no $P_{\text{enc}}$ at all. So one box goes from 1 positive to 8 queries plus up to 13 grid points. These counts are ours, from the configs, not the paper's.

Table. 1 Positives per ground-truth box, RT-DETR against RT-DETRv3-R18. Derived by us from the released configuration [XIA24] and §3.3–3.5 of [WANG24]; the paper prints no such count.

Now the same counts on a toy picture. Start at one-to-one: three objects, three blue boxes, 37 of the 40 queries told "nothing here". Then turn the helpers on one at a time.

query groups (1 = plain one-to-one)

1

one-to-many copies m (0 = off)

0

CNN head: grid points per object k (0 = off)

0

one-to-oneCNN headquery groupsone-to-manyeverything onReset

2. A CNN Head on the Encoder

2.1 What it is

• A PP-YOLOE head [XU22], used unchanged, reads the encoder's three output maps $C_3,C_4,C_5$ in parallel with the decoder (§3.1, §3.3, §4.2); the config puts them at strides 8, 16 and 32 [XIA24].
• It is a dense detector: every grid point of every map predicts a class score and a box, and many grid points per object are trained as positive (one-to-many).
• Which points? Early in training the ATSS assigner [ZHAN19]: take the nearest anchor points on each level, keep those whose IoU with the object is above the mean plus the standard deviation of the candidates. Later the task-aligned assigner of TOOD [FENG21] (§3.3; the code switches at epoch 30 [XIA24]).

The task-aligned assigner ranks the points inside a box by one number:

• $s$: the point's score for the object's class; $\mathrm{I}\mathrm{o}\mathrm{U}$: overlap of the point's predicted box with the object; $\alpha ,\beta$ from the config [XIA24]. The top 13 by $t$ become positives.
• β = 6 makes geometry dominate. A point with s = 0.6 and IoU 0.8 scores 0.6 × 0.262 = 0.157; a surer point with s = 0.9 but IoU 0.6 scores 0.9 × 0.047 = 0.042. The well-placed box wins by almost 4×.
• Losses: varifocal loss for the class (the IoU is the target for positives, [ZHAN21]) and a distribution loss for the box (the code adds a GIoU term [XIA24]). The paper notes that the DETR decoder also uses VFL, so both heads define "class score" the same way (§3.3). VFL is explained in RT-DETR — DETRs Beat YOLOs, Encoder by Encoder.

query groups (1 = plain one-to-one)

1

one-to-many copies m (0 = off)

0

CNN head: grid points per object k (0 = off)

13

one-to-oneCNN headquery groupsone-to-manyeverything onReset

2.2 Why it helps

The encoder only learns through the handful of matched queries that read it. Bolt a dense detector onto the same maps and every grid point near an object gets told what it should encode. The decoder then reads better features, with nothing changed in the decoder. Co-DETR [ZONG22] put dense one-to-many heads on a DETR encoder for the same reason; v3 cites it (§2.3).

2.3 Measured, cost, removed

• Measured: RT-DETR-R18 46.5 → 47.4 AP (+0.9) with this head alone (Table 4, 72 epochs).
• Cost: a whole PP-YOLOE head on three maps, in training only. The paper reports no training time or memory.
• At inference: removed. The code calls the head only when training [XIA24].

3. Perturbed Query Groups

3.1 The idea: jostle each copy

Hungarian matching is a tie-breaker. When five queries sit on the dog with nearly the same cost, one wins and four lose, and the loser today is often the winner tomorrow only by luck. So make several copies of the query set, and jostle each copy's self-attention a little differently. The tie breaks differently in each copy, and over the copies several of the near-duplicates get to be "the one". The paper's aim: "allowing multiple related queries associated with the same target to have a chance of being assigned as positive sample queries" (§3.4).

• The query-selection module produces N sets of object queries $O{Q}_1,\dots ,O{Q}_N$ (§3.4).
• All sets go through one decoder (shared weights), but the groups cannot look at each other. Several independent one-to-one groups through one decoder is the idea of Group DETR [CHEN22], which v3 cites; v3's addition is the random mask inside each group.
• Each group is matched and lossed exactly like RT-DETR. So every group still teaches one box per object, and the deployed set keeps the habit that lets it skip NMS.

3.2 The mask, as printed

Let's write the paper's Eqs. 1–3 for group i, with a random mask $M_i$ from a "mask generator":

• $Q_i,K_i,V_i$: queries, keys and values of the group's object queries (as in Query, Key, Value — Dot-Product Attention from Scratch); $W_i$: the attention weights; ${\tilde{V}}_i$: the mixed features.
• The paper says the mask is multiplied with the attention logits before the softmax (§3.4). It does not give the mask's distribution or rate.

Here is the catch. Multiplying a logit by 0 does not remove the link: $e^0=1$ still takes its share of the softmax. Take one row of three logits (2.0, 1.5, 1.0) and "drop" the middle link:

• before: weights 7.39, 4.48, 2.72 out of 14.59, so the middle link gets 0.307;
• multiplied by 0: logits (2.0, 0, 1.0), weights 7.39, 1, 2.72 out of 11.11, so it still gets 0.090;
• worse, a negative logit multiplied by 0 goes up to 0, so the "dropped" link would gain weight.

3.3 The mask, as the code runs it

The released code [XIA24] does something cleaner. For each perturbed group it draws a fresh uniform number per query pair at every training step, keeps the link if it is above 0.1, and turns a refused link into $-\infty$ added to the logit:

• $d$: the head width (the $\sqrt{d}$ sits inside the standard attention call); $M_i[a,b]$: 1 if query a may look at query b.
• A refused link gets weight exactly 0, and about one link in ten is refused in every perturbed group. The main group is never masked.

NOTE: the paper's Eq. 2 multiplies the logits by the mask, which would not remove a link; the released code removes each query-to-query link outright with probability 0.1, a rate the paper never states. Everything below, the figure included, follows the code.

The figure is the whole decoder mask in training, shrunk to six queries per group. Blue blocks on the diagonal: the groups, blind to each other. Grey: the denoising queries each group carries (RT-DETR — DETRs Beat YOLOs, Encoder by Encoder, [LI22]). Rust dots: links removed in this draw. Move the drop rate or draw again, and only the perturbed groups change. The third tile runs the printed Eq. 2 on toy logits: the weight it leaves on the "removed" links.

drop rate in the perturbed groups

0.1

one-to-one groups (main + perturbed)

4

random draw (a new one every training step)

1

The groups' losses are averaged (Eq. 4 of the paper):

• ${ℒ}_{\text{det}}^{(i)}$: RT-DETR's detection loss on group i (VFL + L1 + GIoU, on every decoder layer); $N$: groups including the main one, 4 for R18 in the code.

Now count winners. Four groups give each object four positives, and on the toy they are not always the same query:

query groups (1 = plain one-to-one)

4

one-to-many copies m (0 = off)

0

CNN head: grid points per object k (0 = off)

0

one-to-oneCNN headquery groupsone-to-manyeverything onReset

3.4 Measured, cost, removed

• Measured: +MGSA alone (the paper's name, multi-group self-attention perturbation) 46.5 → 47.5 AP (+1.0) (Table 4). Two, three and four perturbation branches give 47.9, 48.1 and 48.0 with every other part on (Table 5); three is the paper's choice, and the spread is 0.2 AP, with one number per row and no variance.
• Cost: each group is a full set of 300 queries, and in the code each brings its own denoising queries and its own query-selection heads [XIA24]; the paper describes neither. The decoder's self-attention grows with the square of the total (§5 below).
• At inference: only the main group runs, with no mask.

4. The One-to-Many Set

4.1 Just ask for four

The groups hope the jostling spreads the positives. The third helper simply asks for them: one more query set, through the same decoder, matched to the ground truth repeated m times (§3.5):

• $M$: objects in the image; $𝒞$: RT-DETR's matching cost; ${\widehat{\pi }}^{(m)}$: the same Hungarian solver as before, now on mM targets. Each object gets m different queries, because a one-to-one matching never gives two targets the same query.
• That needs $Q\ge mM$. With the paper's 300 queries and m = 4 it caps at 75 objects per image; with the code's 450, at 112 (our arithmetic).
• The attention inside this set is not masked, but it is still blind to the other groups (the last block of the mask figure).

This is the same trick as H-DETR's hybrid matching [JIA23], which v3 does not cite.

NOTE: the paper sets 300 queries for this set (§4.2); the released configs use 450 [XIA24].

query groups (1 = plain one-to-one)

1

one-to-many copies m (0 = off)

4

CNN head: grid points per object k (0 = off)

0

one-to-oneCNN headquery groupsone-to-manyeverything onReset

4.2 Why not one-to-many everywhere?

Because a network trained one-to-many learns to put several boxes on each object, and then it needs NMS again. The fix keeps two kinds of query in one decoder: the shipped set is trained one-to-one, the helper set one-to-many, and the shared weights learn from both. That split is what H-DETR's hybrid matching rests on [JIA23]; v3 adds it to a real-time DETR and throws the helper set away at the end.

4.3 Measured, cost, removed

• Measured: +O2M-T alone 46.5 → 47.5 AP (+1.0) (Table 4).
• Cost: 450 more queries through the decoder in training (code).
• At inference: removed.

5. The Total Loss and What Ships

The three helpers add up to one loss (Eq. 5 of the paper):

• ${ℒ}_{aux}$: the CNN head; ${ℒ}_{o2o}$: Eq. 4 above, main group included; ${ℒ}_{o2m}$: the one-to-many set (§3.6).

Now flip the figure to inference: every dashed block is gone, and v3 is v1 again.

version◀3 / 5 · v3▶ Nextstage◀7 / 8 · decoder layers▶ Next▶modeinferencetrainingmodel sizeR18 / SR34 / MR50 / LR101 / Xwhole pipelinev1 at inferencev1 in trainingv2 at inferencev3 in trainingv3 at inferencev4 in trainingv4 at inferenceWorld in trainingWorld at inferenceReset

The series ledger, with v3's row:

Table. 2 The series ledger after part 3. Row v3 from §3 and Table 1 of [WANG24] (same parameters, GFLOPs and latency as v1) and the code [XIA24]; rows v1 and v2 from RT-DETR — DETRs Beat YOLOs, Encoder by Encoder and RT-DETRv2 — A Bag of Freebies for the Decoder.

What the helpers cost in training, the paper does not say. Let's count the decoder's queries for R18 with the code's settings:

• 300 object queries and up to 200 denoising queries per one-to-one group, 450 in the one-to-many set: about 5× the queries of RT-DETR in training, and a self-attention matrix with about 24× the entries, plus the PP-YOLOE head. Our count, not a measurement; the paper reports no training time, memory or throughput.

6. Results

All numbers are COCO val2017 AP. Latency is on a T4 GPU with TensorRT FP16, as for RT-DETR; the paper does not restate the batch size or the input size used for timing (§4.2). R18 and R34 train for 120 epochs, R50 and R101 for 72 (§4.2).

6.1 Against v1 and v2

Table. 3 Table 1 of [WANG24]: the three versions share parameters, GFLOPs and latency per backbone. "–": not reported.

• The gain is largest early and small. At 12 epochs R18 goes 38.7 → 41.5 (+2.8 over v1, +1.7 over v2). At 72 epochs: +1.6 / +1.0 / +0.3 / +0.3 over v1 for R18 / R34 / R50 / R101 (§4.3). More positives help most when training is short and the model is small.
• Against v2 at R50 the gain is +0.0 (53.4 against 53.4), and +0.3 at R101 (our subtraction; the paper compares with v2 only at R18 and R34: +1.4 / +0.8 and +0.9 / +0.2 at 72 / 120 epochs).
• "Half the epochs" (§4.3) is close, not equal: v3 at 36 epochs reaches 46.1 (R18) and 48.6 (R34), against RT-DETR's 46.5 and 48.9 at 72.
• The paper reports only AP: no AP_S, AP_M or AP_L, so I cannot tell you which object sizes gained.

NOTE: the headline R50 53.4 and R101 54.6 (Tables 1 and 2, "72 epochs") are the epoch-51 values of the overfitting table (Table 3 of the paper). At epoch 72, without extra data, the same models score 52.9 and 54.2. The printed numbers look like best checkpoints; the paper does not say so.

Table. 4 Table 3 of [WANG24]: without extra data the large models get worse after epoch 51; the paper blames the size of COCO against the size of the model (§4.3).

6.2 What each part buys

Toggle the three ablations. The first view is the one to stare at: three parts worth +0.9, +1.0 and +1.0 alone give +1.6 together, not +2.9. They overlap, which makes sense: all three are ways of handing the same objects more positives.

tableparts (Table 4)groups (Table 5)schedules (Table 1)backbone (schedules)R18R34R50R101partsgroupsschedulesReset

6.3 Against the YOLOs

Table. 5 Selected rows of Table 2 of [WANG24] (YOLOv10 is [WANG24a]). The paper calls these end-to-end speeds (§4.4) but does not say how NMS was timed for the YOLOs that need it; YOLOv10 is NMS-free by design.

The same table as a picture, latency against AP, with v3's points marked:

table◀3 / 5 · v3 paper: Tables 1 and 2▶ Nextmarked version◀3 / 4 · v3▶ Next

Which families to show

YOLOshideshowother DETRshideshowRT-DETR familyhideshow

v1 paper's tablev2 paper's tablev3 paper's tablev4 paper's tablethe RT-DETR line, v1 markedthe RT-DETR line, v2 markedthe RT-DETR line, v3 markedthe RT-DETR line, v4 markedReset

• v3-R18 beats every small YOLO in the paper's table on AP (by 2.0 to 4.4) with 120 epochs against their 300 to 500 (§4.4). R50 beats YOLOv9-C by 0.9 AP and is 1.37 ms faster (10.57 − 9.2; the paper says 1.3 ms, §1).
• Read the rest of the rows too. YOLOv10 is faster at every size (2.5 against 4.6 ms; 4.7 against 6.3; 7.28 against 9.2; 10.7 against 13.5) with far fewer parameters. R34 (50.1) sits below the medium YOLOs (YOLOv10-M 51.1). R101 beats YOLOv10-X by 0.2 AP.
• The authors say it themselves: "we have not yet optimized the overall framework of the RT-DETRv3 detector for lightweight deployment" (§4.4).

7. Limitations

My reading, each point backed by the paper or the code:

• No training cost anywhere. The helpers multiply the decoder's training queries by about 5 (§5); the paper reports no time, memory or throughput.
• Gains shrink with size: +1.6 AP at R18, +0.3 at R50 and R101 over v1, and +0.0 over v2 at R50 (Table 1).
• Best checkpoint, not end of schedule, for R50 and R101 (the NOTE in §6.1), and the large models overfit COCO without extra data.
• Ablations are at R18 only, one number per row, with no variance reported; Table 5's spread (0.2 AP) is within what a second seed could move.
• Paper and code disagree in four places: the mask (product against removal at rate 0.1), the one-to-many set (300 against 450 queries), the R50 groups (3 perturbed against 2 in the config), and the extra selection heads and denoising groups the paper does not describe. The R101 config listed in the README is missing from the repository [XIA24].
• The code is PaddlePaddle only.

8. What Came Next

v3 showed how far you can go by changing only what the network is taught: three helpers, all thrown away, and the deployed network is v1's. RT-DETRv4 — Borrowing a Foundation Model's Eyes [LIAO25] keeps the move, training-only help and nothing extra at inference, but changes both the helper and the base. The helper is a frozen vision foundation model whose features coach the one map every later stage reads. The base is not v3's network at all: it is the D-FINE detector trained with the DEIM recipe on HGNetv2 backbones, so its gains are measured against that base, not against v3. Stay tuned.

9. References

1. [WANG24] Wang, Xia, Lv, Shi. RT-DETRv3: Real-time End-to-End Object Detection with Hierarchical Dense Positive Supervision. WACV 2025, arXiv 2409.08475.
2. [XIA24] RT-DETRv3 official code (PaddlePaddle), github.com/clxia12/RT-DETRv3: configs, rtdetr_transformerv3.py, ppyoloe_head.py.
3. [ZHAO24] Zhao et al. DETRs Beat YOLOs on Real-time Object Detection (RT-DETR). CVPR 2024, arXiv 2304.08069.
4. [LV24] Lv et al. RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer. arXiv 2407.17140.
5. [LIAO25] Liao et al. RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models. arXiv 2510.25257.
6. [CARI20] Carion et al. End-to-End Object Detection with Transformers (DETR). ECCV 2020, arXiv 2005.12872.
7. [JIA23] Jia et al. DETRs with Hybrid Matching (H-DETR). CVPR 2023, arXiv 2207.13080.
8. [CHEN22] Chen et al. Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment. arXiv 2207.13085.
9. [XU22] Xu et al. PP-YOLOE: An evolved version of YOLO. arXiv 2203.16250.
10. [ZONG22] Zong, Song, Liu. DETRs with Collaborative Hybrid Assignments Training (Co-DETR). arXiv 2211.12860.
11. [ZHAN19] Zhang et al. Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection (ATSS). CVPR 2020, arXiv 1912.02424.
12. [FENG21] Feng et al. TOOD: Task-aligned One-stage Object Detection. ICCV 2021, arXiv 2108.07755.
13. [ZHAN21] Zhang et al. VarifocalNet: An IoU-aware Dense Object Detector. CVPR 2021, arXiv 2008.13367.
14. [LI22] Li et al. DN-DETR: Accelerate DETR Training by Introducing Query DeNoising. CVPR 2022, arXiv 2203.01305.
15. [WANG24a] Wang et al. YOLOv10: Real-Time End-to-End Object Detection. arXiv 2405.14458.