RT-DETR — DETRs Beat YOLOs, Encoder by Encoder

A real-time detector has two jobs at once: be right, and be done in a few milliseconds. YOLOs are quick, but they end with NMS, and its time and accuracy both hang on two thresholds. DETRs (DETR — Detection as Set Prediction, No Anchors and No NMS) need no NMS, but they were never quick. RT-DETR [ZHAO24] (Zhao, Lv and colleagues at Baidu and Peking University, CVPR 2024) calls itself "the first real-time end-to-end object detector" (abstract), and on its own benchmark it beats the YOLOs of its day on both counts: 53.1 AP at 108 FPS, about 9.26 ms, for the ResNet-50 model on a T4 GPU (Table 2).

The pipeline, which this whole post hangs on:

• backbone → three maps S3, S4, S5 at strides 8, 16, 32;
• AIFI: one attention layer, on S5 only;
• CCFF: a convolutional fusion of the three maps;
• query selection: the best 300 of the 8,400 positions become the queries;
• decoder: six deformable layers refine 300 boxes; no NMS afterwards.

version◀1 / 5 · v1▶ Nextstage◀1 / 8 · whole pipeline▶ Next▶modeinferencetrainingmodel sizeR18 / SR34 / MR50 / LR101 / Xwhole pipelinev1 at inferencev1 in trainingv2 at inferencev3 in trainingv3 at inferencev4 in trainingv4 at inferenceWorld in trainingWorld at inferenceReset

Step through the stages to see each block's shapes; we meet them in order.

This is part 1 of 5: RT-DETR here, then RT-DETRv2, v3 and v4, and last RT-DETR-World, an open-vocabulary detector from a different group that borrows the name. The thread: after v1, almost every gain comes from parts that exist only in training, so every post ends with the same ledger of what is trained, what ships, and what it costs at inference.

Brush up before you start ;)

• The set loss, Hungarian matching, GIoU, and single-scale deformable attention (§7 there): DETR — Detection as Set Prediction, No Anchors and No NMS. I will only say what RT-DETR changes.
• Focal loss and the α, γ knobs: Understanding Focal Loss —A Quick Read.
• NMS, IoU, AP and the one-stage vocabulary: Object Detection — A Quick Read. Why attention has to watch its token count: Attention Meets Pixels — Windows, Axes and the First Fully Attentional Networks.

1. What NMS Costs

Picture a photographer who shoots 9,000 frames at a party and then sits down to throw away the duplicates. How long that takes depends on how many frames he kept (the confidence threshold) and how alike two frames must be before he calls them duplicates (the IoU threshold). That is NMS. The paper's description (§3.1): drop every box under the confidence threshold, then, whenever two boxes overlap above the IoU threshold, drop the lower-scored one, class by class.

So its time depends on how many boxes survive the first filter. The paper counts them on one image, for an anchor-based YOLOv5 and an anchor-free YOLOv8.

detectorYOLOv5YOLOv8confidence threshold◀1 / 6 · 0.001▶ NextIoU threshold◀3 / 4 · 0.7▶ NextYOLOv5, anchor-baseda strict thresholdReset

• At confidence 0.001, YOLOv5 hands NMS 9,281 boxes and YOLOv8 3,570. That is 2.6× (the text says "three times more", §3.2; the bars say 2.6).
• Raise the threshold to 0.25 and the pile drops to 581 and 381. Fewer boxes, less work.
• The counts come from the bar labels of the July 2023 arXiv version (2304.08069v2, Fig. 2); the CVPR version prints none.

Now the price. Table 1 times YOLOv8 on COCO val2017, on a T4 with TensorRT FP16:

Table. 1 Table 1 of [ZHAO24]: YOLOv8 on COCO val2017. The time is the NMS kernel alone, not the whole post-processing.

• The best AP, 52.9, sits at the slow end. Buying 1.7 AP (51.2 → 52.9) costs 1.3 ms of NMS kernel alone (1.06 → 2.36).
• The kernel alone takes 1.06–2.46 ms. For a detector that runs in 10 ms, that is not small change.
• YOLO papers usually report speed without NMS. So RT-DETR builds an end-to-end speed benchmark (§3.2): every YOLO is timed with NMS attached, at the thresholds that give it its published AP. Keep this in mind when we reach the results.

2. Why DETRs Were Slow

DETR read one stride-32 map. Deformable DETR [ZHU20] made training converge faster by reading several scales, and the price was a much longer token sequence for the encoder. RT-DETR quotes Lin et al. [LIN22] on the bill: in Deformable DETR the encoder takes 49 % of the GFLOPs but brings only 11 % of the AP (§4.2). That is a second-hand number: [LIN22] trained Deformable DETR without its encoder and found the encoder adds 4.9 AP for about 85 GFLOPs (§1 there). It was not measured in this paper.

Let's count the tokens at 640 × 640. Map l has stride $s_l$, so it has $N_l$ positions:

• S3 is 80 × 80, S4 40 × 40, S5 20 × 20. S3 alone is 76 % of the tokens, and it is the map with the least meaning per cell.

Now count the pairs one dense attention layer would score, for one head:

• Attending only within S5 is 441× fewer pairs than one long sequence. Tempting to stop here and say "AIFI is 441× cheaper". It is not, and the figure shows why.
• The encoder RT-DETR replaces was never dense. A deformable encoder layer samples 8,400 tokens × 8 heads × 3 levels × 4 points = 806,400 values. AIFI on S5 scores 400² × 8 heads = 1.28 M pairs. Same order.
• What drops 21× is the number of tokens that go through the layer's projections and feed-forward network: 8,400 → 400.

showcountsmeasuredinput side (px)

640

encoder variant◀6 / 6 · E▶ Next▶the A to E walkAIFI on S5 onlycounts at 1280Reset

At 640 the rust line (deformable) and the blue line (AIFI) are within a factor of 1.6. AIFI grows with the square of S5's tokens, so at 1280 it is 6× the deformable count. Counts, not time; the time is in the next section.

3. The Efficient Hybrid Encoder

3.1 Walking from A to E

The paper walks to its encoder in five measured steps (Fig. 3, Table 3). The test model is DINO-Deformable-R50 with RT-DETR's lighter decoder and data loading, trained 12 epochs. Press play.

showcountsmeasuredinput side (px)

640

encoder variant◀1 / 6 · A▶ Next▶the A to E walkAIFI on S5 onlycounts at 1280Reset

• A: DINO-Deformable-R50 with a lighter decoder and no multi-scale encoder: 43.0 AP, 7.2 ms.
• A → B: one Transformer layer per scale, each scale attending only to itself. +1.9 AP, but latency +54 %.
• B → C: all scales in one sequence. +0.7 AP, +20 % latency.
• C → D: attention within each scale, and a PANet-style path between scales. +0.8 AP and 8 % faster.
• D → D S5: attention on S5 only. +0.4 AP and 35 % faster. Less attention, better accuracy.
• D → E: RT-DETR's encoder. +1.5 AP, 24 % faster than D, 20 % more parameters.

NOTE: E (9.3 ms) is 1.4 ms slower than D S5 (7.9 ms) for +1.1 AP; the paper compares E with D, not with D S5. Its 9.3 ms equals the full RT-DETR-R50 latency of Table 5, so the column looks like whole-model latency, but the paper does not say.

3.2 AIFI: Attention on S5 Only

The paper's Eq. 1 is the whole encoder in three lines:

• AIFI (Attention-based Intra-scale Feature Interaction) is one standard Transformer encoder layer: 8 heads, width 256, FFN 1024 (Table A). 2-D sine-cosine positions are added to the queries and keys.
• The shapes at 640, for R50: S5 is 2048 × 20 × 20 → a 1×1 convolution with BatchNorm → 256 × 20 × 20 → flatten → 400 tokens × 256. Reshape gives F5 back as 256 × 20 × 20. (R101 uses width 384.)
• Why only S5? The paper's argument (§4.2): attention on high-level features relates "conceptual entities"; on low-level features it is unnecessary and risks clashing with the high-level interaction. D S5 vs D in Table 3 is the evidence.

version◀1 / 5 · v1▶ Nextstage◀4 / 8 · AIFI on S5▶ Next▶modeinferencetrainingmodel sizeR18 / SR34 / MR50 / LR101 / Xwhole pipelinev1 at inferencev1 in trainingv2 at inferencev3 in trainingv3 at inferencev4 in trainingv4 at inferenceWorld in trainingWorld at inferenceReset

3.3 CCFF: Fusing the Scales with Convolutions

CCFF (CNN-based Cross-scale Feature Fusion) is a PANet-style path [LIU18]: top-down from F5 to S3, then bottom-up again, with a fusion block wherever two scales meet (Fig. 5). A fusion block takes two adjacent maps, concatenated, and does:

• $x$: the two maps concatenated, 512 channels; the two 1×1 convolutions bring it to 256; ${\mathrm{R}\mathrm{e}\mathrm{p}}^3$ is three RepBlocks in a row (Table A). The two paths are added.
• Upsampling on the way down is nearest ×2; on the way up a 3×3 stride-2 convolution halves the map.
• Out come three maps of 256 channels: 80 × 80, 40 × 40, 20 × 20, flattened into the 8,400 × 256 memory the decoder reads.

3.4 The RepBlock: Two Kernels in Training, One at Inference

A RepBlock comes from RepVGG [DING21]. In training it has two branches, a 3×3 convolution and a 1×1 convolution, each with its own BatchNorm, added. At inference both are linear, so they fold into one 3×3 kernel:

• $\gamma ,\beta ,\mu ,\sigma$: each BatchNorm's scale, shift, running mean and running standard deviation; $\mathrm{p}\mathrm{a}\mathrm{d}$ puts the 1×1 kernel in the centre of a 3×3 one.
• Training gets two paths to learn through; deployment pays for one convolution.
• This is the series' first thing that exists in training and is gone at inference. Remember it: v2, v3 and v4 all play the same move, harder.

4. Query Selection: 300 Hands Up

4.1 Queries from the Image

A teacher with 300 questions can call on 300 fixed seats, whoever sits there, or on the 300 students with their hands up highest. DETR does the first: 100 learned queries, the same for every image. RT-DETR does the second.

• Every one of the 8,400 encoder positions predicts 80 class scores and a box.
• The top 300 by class score are kept. Their features become the content queries (300 × 256); their boxes become the starting reference boxes (300 × 4), from which the position embeddings are made.
• The decoder starts from boxes that already sit on objects.

One more difference from DETR, and readers trip on it. DETR's head is a softmax over 92 entries, the last one ∅, "no object". RT-DETR's head is 80 sigmoids and no ∅: each class is its own yes/no, as in focal loss [LIN17]. "Nothing here" just means every score is low.

4.2 What the Paper Writes, and What the Code Does

Picking by class score has a catch: a position can be sure what it sees and still have a poor box. The paper calls the gap uncertainty (Eq. 2) and puts it into the loss (Eq. 3):

• $\widehat{x}$: an encoder feature; $𝒫,𝒞$: its localisation and classification "distributions"; $\widehat{y}=(\widehat{c},\widehat{b})$ the prediction, $y=(c,b)$ the truth.
• The paper defines neither the norm nor a localisation distribution as a number, so let's read the official v1 code [LYU23]. The July 2023 arXiv version called this IoU-aware query selection, citing VarifocalNet [ZHAN21], and the code does exactly that: the top 300 encoder picks are matched to the objects and trained with the varifocal loss, the IoU as the target.

For a pick matched to an object whose box it predicts with IoU $q$, and p the predicted probability of the true class:

• ${ℒ}^{+}$: the true class, a cross-entropy with the soft target q, weighted by q itself. ${ℒ}^{-}$: every other class, and every unmatched pick, pushed to 0 focal-style, α = 0.75, γ = 2 (Table A).

Where is ${ℒ}^{+}$ lowest? Let's take the derivative and set it to zero:

• That is the whole trick. The class score is trained to equal the IoU of its box. A pick with a sloppy box (q = 0.3) is told "0.3", not "1". So "top 300 by class score" also leans toward the picks with good boxes. Table 4 below measures how much.
• Numbers, q = 0.8: the loss is 0.400 at p = 0.8 and 0.555 at p = 0.5. A negative at p = 0.5 costs 0.75 · 0.25 · ln 2 = 0.130, against 0.693 unweighted.

showthe training targetpicking the top K (toy)IoU of the matched box, q

0.8

α (negatives)

0.75

γ (negatives)

2

Toy: picking the top K

how much the score tracks the IoU

0

K picked

60

top K, toytop K, score tracks IoUa poor box, q = 0.3Reset

Slide q: the minimum of the left curve follows it. The dashed curve is the usual target of 1; it keeps falling all the way to p = 1, whatever the box looks like.

4.3 Does It Pick Better?

Table. 2 Table 4 of [ZHAO24]: RT-DETR-R50, 12 epochs, shares over the 300 picks.

• +0.8 AP, and more than twice as many picks with a good box: 0.30 → 0.67.

NOTE: Table 4 prints these shares with a "%" and §5.3 writes "0.82% vs 0.35%". They are fractions: 82 % and 35 %. The 138 % and 120 % of §4.3 count dots in Fig. 6, not shares, so they do not match the 134 % and 123 % you get from this table.

The same idea as a toy: made-up positions, picked by score. Turn up how much the score tracks the IoU and watch the picks move into the shaded corner.

showthe training targetpicking the top K (toy)IoU of the matched box, q

0.8

α (negatives)

0.75

γ (negatives)

2

Toy: picking the top K

how much the score tracks the IoU

0

K picked

60

top K, toytop K, score tracks IoUa poor box, q = 0.3Reset

5. The Decoder

5.1 Reading 96 Points, Not 8,400

Each of the six decoder layers does self-attention among the 300 queries, then multi-scale deformable cross-attention into the 8,400-token memory, then an FFN. DETR — Detection as Set Prediction, No Anchors and No NMS §7 wrote single-scale deformable attention. The multi-scale version adds one sum, over the three maps:

• $z_q$: the query; $b_q=(c_x,c_y,w,h)$: its reference box; $x^l$: map l; M = 8 heads, K = 4 points; $A_{mlqk}$: a weight, softmaxed over all 12 points of the head across the three maps; $W_m,W_m^{\prime }$ as in DETR's version.
• Per query and layer: 8 × 3 × 4 = 96 reads. For 300 queries, 28,800, against 300 × 8,400 = 2.52 M pairs for dense cross-attention.

Where are the points? Deformable DETR offset them around a point. RT-DETR has a whole box, so the offsets are relative to the box:

• $\Delta p_{mlqk}$: a learned offset, predicted from $z_q$. At the start of training (in the code) head m points at angle 2πm/8, stretched so its ray ends on the box's edge: the four side heads aim at the edge midpoints, the four diagonal ones at the corners. Point k sits k/4 of the way out along that ray. A big box reads widely; a small box reads closely.

box centre

(0.42, 0.5)

box width

0.3

box height

0.36

map◀2 / 3 · S4 · 40 × 40▶ Nextrefinement steps (toy)

0

▶a small box on S5a small box on S3refine three timesReset

• Shrink the box to 0.08 × 0.1 at the starting centre (the presets): on S3 its 32 points read 70 of the 6,400 cells, on S5 only 12 of 400. Each map gives the query a different grain of the same region.

5.2 Refining the Box, Layer by Layer

After every layer a small MLP predicts a correction, added in logit space:

• $b_0$: the box from query selection; ${\sigma }^{-1}$ the logit. Working in logit space keeps centre and size between 0 and 1. In training the box handed to the next layer is cut from the gradient; the decoder's training follows DINO [ZHAN22] almost exactly (appendix §1).

5.3 A Decoder You Can Cut Short

Every layer has a class and box head, so every layer can give the answer. At inference you can stop at layer n and skip the rest, with no retraining (§4.4).

trained model◀3 / 7 · R50, 6 layers▶ Nextstop after layer

6

stop at layer 5scaled modelsseven layersReset

• R50 trained with 6 layers: 53.1 AP at 9.3 ms. Stop after layer 5: 53.0 AP at 8.8 ms. Each layer costs ≈ (9.6 − 7.0) / 6 ≈ 0.43 ms (Table 5, my arithmetic).
• Seven layers train worse than six (52.6 vs 53.1).

NOTE: Table 5 has one latency column for all four trained models; the paper does not say which one was timed.

6. Matching and Losses

Matching is one-to-one with the Hungarian algorithm, as in DETR — Detection as Set Prediction, No Anchors and No NMS. The cost changes: with sigmoid classes, the class term becomes focal-style (α = 0.25, γ = 2 for matching):

• $C_{cls}$ is cheap when the query is sure of the true class. Weights 2 / 5 / 2 for matching; the loss uses 1 / 5 / 2, with the varifocal loss of §4 as its class term (Table A).
• The loss is applied to the last layer, to every earlier layer, and to the encoder's top 300.
• Denoising queries. In training, noised copies of the true boxes and labels go through the decoder next to the 300 queries, and the decoder learns to repair them (DN-DETR [LI22], DINO [ZHAN22]). Up to 200 of them, training only.
• Output. 300 queries × 80 sigmoid scores; the best 300 (query, class) pairs are kept. One query can show up twice with two classes. No NMS.

version◀1 / 5 · v1▶ Nextstage◀7 / 8 · decoder layers▶ Next▶modeinferencetrainingmodel sizeR18 / SR34 / MR50 / LR101 / Xwhole pipelinev1 at inferencev1 in trainingv2 at inferencev3 in trainingv3 at inferencev4 in trainingv4 at inferenceWorld in trainingWorld at inferenceReset

Training (appendix §1 and Table A): ResNet-vd backbones [HE15, HE18] pretrained on ImageNet, AdamW at lr 1e-4 (backbone 1e-5), 72 epochs on COCO, batch 16 on 4 V100s.

7. Results

Protocol, for every number below: T4 GPU, TensorRT FP16, batch size 1, 640 × 640, COCO val2017. YOLO times include NMS at the thresholds that give their published AP. Latency is 1000 / FPS, my arithmetic, because the paper prints FPS.

Table. 3 From Table 2 of [ZHAO24]. DINO-Deformable-DETR was timed by the authors at (800, 1333); the other DETRs of that table were not timed.

• R50 against the L YOLOs: +0.2 AP over YOLOv8-L at 9.26 ms instead of 14.08; +0.3 AP over YOLOv6-L, the fastest of them, at 9.26 ms instead of 10.10.
• R101 against the X YOLOs: +0.4 AP over YOLOv8-X at 13.51 ms instead of 20.00.
• Against its own family: +2.2 AP over DINO-Deformable-DETR-R50, and about 21× faster (108 vs 5 FPS).

The smaller sizes and the Objects365 pretraining (the appendix, Tables B and C), with the S and M YOLOs they are compared against:

Table. 4 Tables B and C of [ZHAO24], with Table 2's R50 and R101. The scaled R50's 36 M counts its parameters in training (Table B marks it †). Objects365 first: 12 epochs on Objects365, then 60 (R18) or 24 (R50, R101) on COCO. The July 2023 arXiv version (2304.08069v2, Table 2) also had RT-DETR-L and -X on an HGNetv2 backbone, 53.0 AP at 114 FPS and 54.8 AP at 74 FPS; the CVPR version dropped them.

The scaled models are the decoder dial again, on lighter networks:

trained model◀5 / 7 · scaled R50▶ Nextstop after layer

3

stop at layer 5scaled modelsseven layersReset

NOTE: the appendix says the scaled R50 with 5 layers beats YOLOv8-M by "0.9% AP"; Table B gives 52.1 against 50.2, which is 1.9. I use the table.

Limitations. The one the paper states (§6): small objects. R50's AP_S of 34.8 is 0.5 below YOLOv8-L's 35.3, and R101's 36.0 is 0.9 below YOLOv7-X's 36.9. What else I read in the tables, my own reading:

• Every speed is one GPU (T4), one precision, batch 1. No CPU or edge numbers.
• The YOLOs carry NMS at the thresholds that reproduce their published AP (for YOLOv8-L, 52.9 AP is Table 1's conf 0.001 row): the paper's point, but it makes them slower than their own papers report.
• Not on FLOPs: R50's 136 GFLOPs is more than YOLOv7-L's 104. The win is measured latency.
• 300 queries are a hard cap on detections per image.

8. What Ships and What Stays Behind

RT-DETR is the baseline of this series, and it already trains with parts it throws away:

Table. 5 RT-DETR, part by part, from Tables 2, 5 and A of [ZHAO24].

In one row, the way the series ledger carries it (later posts add a row per version):

Table. 6 The series ledger after part 1.

The one-line takeaway: RT-DETR spends attention only where it pays (one layer on the 20 × 20 map), picks its queries from the image, and needs no NMS, so its measured end-to-end speed beats YOLOs that must pay for NMS.

9. What Came Next

Fifteen months later the same group published RT-DETRv2 — A Bag of Freebies for the Decoder. It keeps this network. The decoder may read a different number of points per scale, or the nearest cell instead of blending four, for runtimes that lack the blended read; and the recipe switches augmentation off for the final epochs and gives each size its own backbone learning rate. The released models cost what the ones above cost. The two training changes add nothing at inference; the two decoder options are dials for deployment, and turning them costs a little AP. Stay tuned ;)

10. References

1. [ZHAO24] Zhao, Lv, Xu, Wei, Wang, Dang, Liu, Chen. DETRs Beat YOLOs on Real-time Object Detection. CVPR 2024, arXiv 2304.08069 (v3; the July 2023 version is 2304.08069v2).
2. [CARI20] Carion et al. End-to-End Object Detection with Transformers. 2020, arXiv 2005.12872.
3. [ZHU20] Zhu et al. Deformable DETR: Deformable Transformers for End-to-End Object Detection. ICLR 2021, arXiv 2010.04159.
4. [LIN22] Lin et al. D²ETR: Decoder-Only DETR with Computationally Efficient Cross-Scale Attention. 2022, arXiv 2203.00860.
5. [LIU18] Liu, Qi, Qin, Shi, Jia. Path Aggregation Network for Instance Segmentation. 2018, arXiv 1803.01534.
6. [DING21] Ding et al. RepVGG: Making VGG-style ConvNets Great Again. 2021, arXiv 2101.03697.
7. [ZHAN21] Zhang, Wang, Dayoub, Sünderhauf. VarifocalNet: An IoU-aware Dense Object Detector. 2021, arXiv 2008.13367.
8. [LIN17] Lin, Goyal, Girshick, He, Dollár. Focal Loss for Dense Object Detection. 2017, arXiv 1708.02002.
9. [LI22] Li et al. DN-DETR: Accelerate DETR Training by Introducing Query DeNoising. 2022, arXiv 2203.01305.
10. [ZHAN22] Zhang et al. DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection. 2022, arXiv 2203.03605.
11. [HE15] He, Zhang, Ren, Sun. Deep Residual Learning for Image Recognition. 2015, arXiv 1512.03385.
12. [HE18] He et al. Bag of Tricks for Image Classification with Convolutional Neural Networks. 2018, arXiv 1812.01187.
13. [LYU23] Lv et al. RT-DETR official code, v1 implementation: github.com/lyuwenyu/RT-DETR, folder rtdetr_pytorch (commit df2a96c).