RT-DETRv2 — A Bag of Freebies for the Decoder

RT-DETR — DETRs Beat YOLOs, Encoder by Encoder made a DETR real-time. In July 2024 the same group published a four-page report, RT-DETRv2 [LV24], whose title already tells you the plan: an "Improved Baseline with Bag-of-Freebies". The network stays the same. What changes is how the decoder reads the feature maps and how the model is trained. The pipeline is the one from part 1, with one block lit up: backbone → AIFI on S5 → CCFF fusion → top-300 query selection → deformable decoder (changed) → 300 boxes, no NMS.

version◀2 / 5 · v2▶ Nextstage◀7 / 8 · decoder layers▶ Next▶modeinferencetrainingmodel sizeR18 / SR34 / MR50 / LR101 / Xwhole pipelinev1 at inferencev1 in trainingv2 at inferencev3 in trainingv3 at inferencev4 in trainingv4 at inferenceWorld in trainingWorld at inferenceReset

Step the version back to v1 and forward to v2: only the decoder changes, and the readout gives the same speed for both. Switch to training to see what v2 adds only while the model learns.

Brush up before you start ;)

• The hybrid encoder, query selection and the multi-scale deformable decoder: RT-DETR — DETRs Beat YOLOs, Encoder by Encoder. This post changes one symbol of its decoder equation.
• Single-scale deformable attention, where the bilinear read first shows up: DETR — Detection as Set Prediction, No Anchors and No NMS §7.
• Where the phrase "bag of freebies" comes from: YOLOv4 — Version 1: Bag of Freebies (in YOLOv4's definition [BOCH20]: things that change only the training, so they cost training time but nothing at inference).

1. What v1 Left on the Table

Let's put numbers on it first. In v1 every decoder query reads, per layer, 8 heads × 3 scales × 4 points = 96 points, whatever the scale [ZHAO24]. At 640 × 640 the three maps are 80 × 80, 40 × 40 and 20 × 20: 6,400, 1,600 and 400 positions. Four looks at a map with 6,400 positions and four looks at a map with 400 are treated as the same budget.

• The same number of points on every scale. The fine stride-8 map and the coarse stride-32 map carry different things, and v1 cannot spend its looks differently on them.
• A read that needs one particular operator. Each point lands between cells, so it is read by bilinear interpolation (the grid_sample operator in PyTorch). The v2 report names this as the one operator RT-DETR needs and YOLO detectors do not, and says it limits where the model can be deployed (§2.1).
• One training recipe for every size. v1's paper gives one backbone learning rate (1e-5) and one augmentation that runs to the last epoch [ZHAO24], and its released configs use that 1e-5 for ResNet-18, -34 and -50 alike [LYU24]. The v2 report calls this sharing of optimizer settings across sizes sub-optimal (§2.2).

v2 answers with four changes, which come in two pairs:

• Two dials for deployment: a separate number of points per scale, and an optional nearest-cell ("discrete") read.
• Two training freebies: augmentation switched off for the last epochs, and a backbone learning rate per model size.

Here is the ledger this post fills in, one row per change. Keep an eye on the last column: that is where "free" is either true or not.

Table. 1 The four changes of [LV24], what each does in training, what is deployed, and what it costs at inference. AP numbers from Tables 2–4 of [LV24]; "speed not reported" because the paper times none of these settings.

2. Points per Scale

2.1 What v1 does, and the one symbol v2 changes

Here is the multi-scale deformable cross-attention of part 1 for one query $q$ and one head $m$, written with the change already in. In v1 the inner sum runs to the same $K=4$ on every scale; v2 replaces $K$ by $n_l$, one count per scale $l$:

• $X_l$: the feature map of scale $l$ (L = 3); $p_q$: the centre of the query's reference box, $(w,h)$ its size; ${\Delta }_{mlqk}$: a learned offset; $A_{mlqk}$: a learned weight; $V_m$: the head's value projection. This is Deformable DETR's form [ZHU20], in the box-relative variant RT-DETR uses.
• $n_l$ enters in three places: the length of the inner sum (cost grows with $n_l$), the offset scale $1/(2n_l)$, and the softmax, which runs jointly over all ${\sum }_l{n}_l$ points of a head, across scales. Add a point on one scale and the weights on the other scales shrink.
• The offset scale keeps the reach fixed. The code starts point $k$ at $k$ unit steps along one of 8 directions (one per head), so the last point, $k=n_l$, sits at $n_l/(2n_l)=$ half the box size whatever $n_l$ is [LYU24].

2.2 Why it makes sense

Think of three photos of the same street: one sharp and large, one medium, one a thumbnail. Looking four times at each is a fixed habit, not a plan. The paper's argument (§2.1) is that the scales carry different kinds of information, so a per-scale count makes the extraction "more flexible and efficient".

2.3 The point budget

Let's count. Every query, in every decoder layer, reads $M{\sum }_l{n}_l$ points, so per image:

• $M=8$ heads, $N_q=300$ queries, $D$ the decoder layers run at inference (3 for v2-S).
• v2-S at 4, 4, 4: $8\times 12\times 300\times 3=$ 86,400 points. At 3, 2 and 1 per scale: 64,800, 43,200 and 21,600. These are the four rows of Table 3 of [LV24], and the arithmetic matches each one.

Table. 2 Fewer sampling points on RT-DETRv2-S, Table 3 of [LV24] (COCO val2017). No timings are given for these rows.

• A quarter of the points costs 0.6 AP (47.9 → 47.3). On S, the decoder loses little with far fewer looks than v1 gave it, which is what makes the count a real dial.

Play with the sliders under "Point budget" below. Equal counts on the S model show Table 3's AP. Anything else says "not measured", because that is the truth:

sample point (u, v) on the map

(0.45, 0.4)

readbilinear readnearest-cell read

Point budget

points on the 80 × 80 map

1

points on the 40 × 40 map

1

points on the 20 × 20 map

1

model◀1 / 5 · S (R18)▶ Next

bilinear readnearest-cell reada quarter of the pointsunequal pointsReset

2.4 What it costs, and what ships

• The idea the paper argues for was never tested. Unequal counts per scale (say 2, 3, 4) are the whole motivation of §2.1, but every row of Table 3 is uniform, and every released v2 model uses 4, 4, 4, exactly v1's setting [LYU24]. So what was measured is "fewer points everywhere", not "different points per scale".
• No speed. Table 3 has no latency column, and the repository says the speed effect depends on the device and software [LYU24]. So I will not write "faster" here: fewer points means fewer reads, counted, not timed.

3. Reading Between Cells

Before the second dial, we need the thing it removes. A sampling point $s$ is a fraction like (0.37, 0.52) of the image, and almost never lands on a cell centre. So how do you read a map between its cells?

Think of reading a ruler between two ticks: if the needle is a tenth of the way from 1 to 2, you read 1.1. The bilinear read does the same in two directions.

Let's devise it step by step. First turn the normalised position $u\in [0,1]$ into a cell index. Cell $i$'s centre sits at $u=(i+0.5)/W$ (the align_corners=False convention), so:

and the same along the other axis: $y=vH-0.5$, $y_0=\lfloor y\rfloor$, $b=y-y_0$. Then the read is a weighted sum of the four surrounding cells:

• The four weights add to 1, and a neighbour outside the map counts as 0.
• Worked example: a map 4 cells wide, $u=0.4$. Then $x=1.6-0.5=1.1$, so $a=0.1$: the read is 0.9 of cell 1 and 0.1 of cell 2 along x. Nudge $u$ and the weights move smoothly.

That smoothness is the whole point. Differentiate the read with respect to the position along x:

• It is the difference between neighbouring cells, mixed by $b$. It is non-zero wherever neighbouring cells differ, so the loss can tell every sampling point which way to move, and the gradient flows back through $s_{mlqk}$ into the offsets ${\Delta }_{mlqk}$ of Eq. 1. This is how a deformable decoder learns where to look.

In the figure, the default point sits at x = 2.2, y = 1.9 on a toy 6 × 6 map. The four weights are 0.08 / 0.02 / 0.72 / 0.18, the value read is 0.766, and the slope along x is −0.23. Drag the point and watch the blue arrow: that is where the slope pushes it. On the right, the bilinear value along the row is a chain of straight segments, and the rust tangent is its slope at the point.

sample point (u, v) on the map

(0.45, 0.4)

readbilinear readnearest-cell read

Point budget

points on the 80 × 80 map

4

points on the 40 × 40 map

4

points on the 20 × 20 map

4

model◀1 / 5 · S (R18)▶ Next

bilinear readnearest-cell reada quarter of the pointsunequal pointsReset

NOTE: DETR — Detection as Set Prediction, No Anchors and No NMS named the bilinear read when it introduced deformable attention; this is where the series writes it out. Per point it costs four reads and a weighted sum.

4. Discrete Sampling

4.1 What v2 does

v2 adds an optional discrete sampling (§2.1 of [LV24]): round the point to a whole cell and read that cell, nothing else. With the convention above, the nearest cell centre is:

• One read, weight 1. In the worked example ($x=1.1$) that is cell 1.
• The derivative is zero almost everywhere. Picture a thermometer that only shows whole degrees: warm it a little and the display does not change, so you cannot tell which way you moved it. Move the point inside a cell and the read stays the same; at the cell border it jumps.
• Zero slope means no training signal reaches the offsets. So in discrete mode the paper freezes the layer that predicts the offsets (§2.1). Attention weights, value projections and everything else still train.

Switch the figure to the nearest-cell read. The four highlights become one, the row turns into a staircase, the tangent goes flat and the slope readout shows 0:

sample point (u, v) on the map

(0.45, 0.4)

readbilinear readnearest-cell read

Point budget

points on the 80 × 80 map

4

points on the 40 × 40 map

4

points on the 20 × 20 map

4

model◀1 / 5 · S (R18)▶ Next

bilinear readnearest-cell reada quarter of the pointsunequal pointsReset

NOTE: the released code computes the cell as round(u·W) rather than ⌊x + 0.5⌋, which reads the right or lower neighbour whenever the fractional part of u·W is 0.5 or more (in the example, cell 2 instead of 1) [LYU24]. The fine-tune stage below runs the same rule, so the model is tuned with the read it is deployed with.

4.2 Why it makes sense

• Bilinear reading exists to make offsets learnable. Once training has put the points where they belong, nothing at inference needs that slope any more.
• Snapping keeps the learned placement, replaces four reads and a weighted sum by one indexed read, and drops the one operator RT-DETR needed and YOLOs did not.
• The paper names no runtime. For context: ONNX first defines GridSample in opset 16 [ONNX22], and ONNX-TensorRT added GridSample with its TensorRT 8.5 GA release (November 2022) [NVID22]. The repository times the normal models with TensorRT ≥ 8.5.1 and says the discrete models run on TensorRT 8.4 "or even older versions" [LYU24].

4.3 The schedule

Train with the bilinear read first, then swap in the nearest-cell read and fine-tune with the offset layer frozen; deploy the fine-tuned model (§2.1). The paper's setting is 6× with bilinear, then 1× discrete (§3.1).

NOTE: the released fine-tune configs are 36 epochs for S and M* and 12 for M and L, each starting from the released bilinear checkpoint [LYU24], not one fixed 1× everywhere.

4.4 Measured effect, and what it costs

Table. 3 Discrete sampling, Table 4 of [LV24] against the bilinear models of its Table 2 (COCO val2017). ΔAP50 is printed in Table 4; ΔAP is my subtraction. No X row and no timings.

• The paper says discrete sampling "does not cause a noticeable reduction" in AP50 (§3.4), and in AP50 that holds: 0.1 to 0.4. In the full AP (IoU 0.50:0.95) the drop is 0.5 to 0.7, which the text does not mention. Snapping to a cell costs precision, and AP at high IoU is where you see it.
• So the bill is: 0.5–0.7 AP, an extra fine-tune stage, and frozen offsets. What you buy is a simpler operator. Whether it is also faster is not measured; the repository again says it depends on the device [LYU24].

5. Two Training Freebies

These two never touch the deployed graph. They are freebies in YOLOv4's exact sense [BOCH20].

5.1 Dynamic augmentation

• v1: colour distortion, zoom-out, random crop and flip [ZHAO24], plus a multi-scale input in the released Paddle configs (each batch resized to a size between 480 and 800 px) [LYU24], for every epoch.
• v2: keep them early, and switch off the colour distortion, zoom-out, crop and multi-scale input for the last two epochs (§2.2).
• Why: early on the model generalises poorly and heavy distortion keeps it from memorising; at the end, the training images should look like the test images (640 px, natural colours), so its last steps tune to the real distribution. Think of training in the rain for months, then doing the final rehearsal in the stadium where you will play.

Table. 4 When the augmentation stops in the released configs [LYU24] (epochs counted from 0). The paper says "the last two epochs"; the configs use one to three. The S configs never use the multi-scale input at all.

5.2 A backbone learning rate per size

• v1: base lr 1e-4 and backbone lr 1e-5 in the paper's one hyperparameter table, written for R50 [ZHAO24]; the released configs keep 1e-5 for R18, R34 and R50 [LYU24].
• v2: the detector stays at 1e-4; the backbone gets a learning rate that falls with its size (Table 1 of [LV24]):

Table. 5 Scale-adaptive learning rates, Table 1 of [LV24]. M* (R50-m) is not in the table; its config uses 1e-5 [LYU24].

• Why (§2.2): a small ImageNet backbone gives weaker detection features, so let it move more; a large one already gives good features, so leave it nearly where it is.
• Our reading, not the paper's: v1's own released R101 config already used 1e-6 [LYU24], and L stays at 1e-5. So in practice the change is real for S (10×) and M (5×), and those are the sizes that gain most in the results below.

Neither freebie is ablated on its own. Their share of the gains cannot be separated.

6. Results

Protocol, from Table 2 of [LV24] (caption and header) and its §3.2: T4 GPU, TensorRT FP16, batch size 1, 640 × 640, COCO val2017. The paper prints FPS; the latency in ms is 1000 / FPS, my arithmetic.

Table. 6 RT-DETR against RT-DETRv2, Table 2 of [LV24]. Parameters and FPS are the same for v1 and v2. GFLOPs are not in the v2 paper; they come from the v1 paper [ZHAO24] and the repository [LYU24].

The same table as a plot of latency against AP. v1 and v2 sit at the same latency for every size, so each v2 point lands straight above its v1 point:

table◀2 / 5 · v2 paper: Table 2▶ Nextmarked version◀2 / 4 · v2▶ Next

Which families to show

YOLOshideshowother DETRshideshowRT-DETR familyhideshow

v1 paper's tablev2 paper's tablev3 paper's tablev4 paper's tablethe RT-DETR line, v1 markedthe RT-DETR line, v2 markedthe RT-DETR line, v3 markedthe RT-DETR line, v4 markedReset

• Same speed, same size: identical parameters and FPS at every size. That is the paper's "without loss of speed" (§3.3), and for the released models it holds.
• The gain shrinks as the model grows: +1.4 AP for S down to 0.0 for X. The gain is largest where the backbone is weakest; how much of it each change brings, the paper does not separate.

Now the honest part. Three things you should know before quoting this table:

• Longer schedules (our reading). v1 trained every size for 72 epochs. The released v2 configs train S and M for 120 epochs and M* for 84; L and X stay at 72 [LYU24]. The paper never mentions this. The biggest gains (S, M) sit exactly where the schedules got longer, so part of them may be extra training, not the freebies.
• No unequal point counts were tested (§2.4), and every released model reads 4, 4, 4. The gains above come from training, not from the decoder change.
• The labels. In the paper M = R34 and M = R50-m (the paper writes "ResNet50" for both M and L; R50-m is the repository's name for M: a ResNet-50 with a halved fusion block, read out after 3 decoder layers); the repository's README swaps M and M [LYU24]. I use the paper's labels and always give the backbone.

NOTE: S is 47.9 / 64.9 in the paper. The repository later lists a rerun of S at 48.1 / 65.1 [LYU24]. I quote the paper's number everywhere in this post.

7. What Ships and What Stays Behind

The series ledger from RT-DETR — DETRs Beat YOLOs, Encoder by Encoder §8, with v2's row added:

Table. 7 The series ledger after part 2. AP costs from Tables 3 and 4 of [LV24].

In one line: v2 changes nothing you pay for at inference unless you choose to. Two training freebies buy +0.3 to +1.4 AP for S to L (with the schedule caveat), and two deployment dials trade a little AP for fewer reads or a simpler operator.

8. What Came Next

v2 left the matching alone: in training, each object still teaches exactly one query per decoder layer, and every other query sitting on the same object hears "nothing here". RT-DETRv3 — More Positives Without Paying at Inference attacks that. It adds branches that hand each object many positives while training (a dense CNN head on the encoder, perturbed query groups, a one-to-many query set) and throws all of them away before deployment, so the inference graph stays v1's. Stay tuned.

9. References

1. [LV24] Lv, Zhao, Chang, Huang, Wang, Liu. RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer. 2024, arXiv 2407.17140.
2. [ZHAO24] Zhao, Lv, Xu, Wei, Wang, Dang, Liu, Chen. DETRs Beat YOLOs on Real-time Object Detection. CVPR 2024, arXiv 2304.08069.
3. [ZHU20] Zhu, Su, Lu, Li, Wang, Dai. Deformable DETR: Deformable Transformers for End-to-End Object Detection. ICLR 2021, arXiv 2010.04159.
4. [BOCH20] Bochkovskiy, Wang, Liao. YOLOv4: Optimal Speed and Accuracy of Object Detection. 2020, arXiv 2004.10934.
5. [LYU24] Lv et al. RT-DETR / RT-DETRv2 official implementation, github.com/lyuwenyu/RT-DETR (folder rtdetrv2_pytorch, commit df2a96c).
6. [ONNX22] ONNX operator changelog, GridSample-16 (opset 16).
7. [NVID22] ONNX-TensorRT changelog, TensorRT 8.5 GA release, 2022-11-02.