RT-DETRv4 — Borrowing a Foundation Model's Eyes

In RT-DETR — DETRs Beat YOLOs, Encoder by Encoder, AIFI is the one attention layer the real-time DETR kept: it reads the 20 × 20 map S5 and writes F5, the map that the fusion, the query selection and the decoder all depend on. RT-DETRv4 [LIAO25] starts from an awkward number about that layer. In a well-trained baseline, AIFI receives about 0.5 % of the training gradient (Tab. C of the paper). The layer every later stage leans on hears almost nothing from the loss.

The fix fits in three steps:

1. Bring a teacher. A frozen DINOv3 ViT-B/16 [SIME25], distilled from a larger DINOv3 that learned from about 1.7 billion images without labels, looks at the same training image.
2. Point F5 the teacher's way. A linear projector and a cosine loss pull each of F5's 400 cells toward the teacher's feature at the same place.
3. Keep the pull steady. The loss weight λ is retuned once per epoch so that AIFI's share of the gradient stays in a target band.

Then the teacher, the projector and the loss are all thrown away, and the network you deploy is the one you started with.

version◀4 / 5 · v4▶ Nextstage◀4 / 8 · AIFI on S5▶ Next▶modeinferencetrainingmodel sizeR18 / SR34 / MR50 / LR101 / Xwhole pipelinev1 at inferencev1 in trainingv2 at inferencev3 in trainingv3 at inferencev4 in trainingv4 at inferenceWorld in trainingWorld at inferenceReset

The dashed blocks at the top are v4's: the DINOv3 teacher and the projector hanging off AIFI. Switch to inference and both disappear. Step back to v3 and notice how much else changes too; that comes first.

Brush up before you start ;)

• AIFI, F5 and CCFF: RT-DETR — DETRs Beat YOLOs, Encoder by Encoder §3. I will not repeat them.
• Distillation, a student copying a teacher: DeiT — Training a Vision Transformer on ImageNet Alone (there a convnet teaches a ViT; here it is the other way round).
• Patch tokens, the teacher's language: An Image Is Worth 16×16 Words — ViT, Patch by Patch. One of the teachers tried is an MAE: Masked Autoencoders — Hide 75 %, Encode the Rest.
• Training-only branches, last time: RT-DETRv3 — More Positives Without Paying at Inference.

1. Whose Network Is This?

Most readers will assume v4 = v3 + a teacher. It is not. v4 comes from the same group as v1 (Yian Zhao, first author of the CVPR version of RT-DETR, is joint first author and project leader of v4), but the network it trains comes from two other lines of work:

• D-FINE [PENG24] keeps the RT-DETR frame and changes the box head: instead of predicting four coordinates, each decoder layer predicts a probability distribution over how far each box edge should move, and refines it layer by layer (FDR). The last layer's distributions then teach the earlier layers (GO-LSD, a self-distillation). D-FINE also fixes the backbones and sizes: HGNetv2 B0 / B2 / B4 / B5 for S / M / L / X (HGNetv2 itself had already appeared in the July 2023 arXiv version of the RT-DETR paper, 2304.08069v2).
• DEIM [HUAN24] is a training recipe for DETRs. One-to-one matching gives each object a single positive query, the same complaint as in RT-DETRv3 — More Positives Without Paying at Inference. DEIM answers it with data instead of extra branches: augmentation packs more objects into each image (Dense O2O), and a Matchability-Aware Loss handles the many low-quality matches that this creates. It roughly halves the schedule.

Table. 1 Where each part of RT-DETRv4 comes from, per the paper's implementation details (App. A) and the released configs: each size includes D-FINE's model file, and the repository's v4 base config differs from its DEIM base config only by the extra loss term. App. A also names RT-DETRv2 as a source of refinements without saying which.

So the honest baseline is DEIM-trained D-FINE, called DEIM in the paper's tables. The deployed v4-S/M/L/X has exactly DEIM's parameters (10 / 19 / 31 / 62 M), GFLOPs and latency (Tab. 1). v4's own gain is measured against it: +0.8 / +1.0 / +0.7 / +0.5 AP. The jump over v3 at the same letter (L: 53.4 → 55.4) mixes a new backbone, a new head and a new recipe, and says little about the teacher.

NOTE: v4's "L" is not v2's or v3's "L". RT-DETRv3-R50 is a ResNet-50 model with 42 M parameters; RT-DETRv4-L is HGNetv2-B4 with 31 M.

2. The Semantic Bottleneck

Why would AIFI starve? Follow the gradient backwards. The detection loss sits on the decoder's outputs. On its way back to AIFI it passes through the decoder, the query selection and CCFF, and most of it lands in the backbone instead: the backbone takes 86–88 % of the gradient's L1 norm in every setting the paper measures (Tab. C). The authors name this the "Semantic Bottleneck" (§3.2).

The numbers for the DEIM-L baseline (Tab. C):

• AIFI's share is 0.64 % in epochs 1–12 and 0.45 % by epochs 36–58; 0.51 % overall.
• It falls as training goes on, so the layer learns least exactly when the rest of the detector settles.

The paper's deeper motivation is the information bottleneck [TISH00]: a good feature keeps what predicts the labels and drops the rest of the image, while a detection loss only rewards the first half (§3.1). A foundation model's features are offered as a target that has already dropped much of the clutter. That is an argument, not a measurement; the measurements are the two tables in 3.1 below.

Here is the picture I keep in mind. You are learning to draw from a handful of labelled sketches, and every correction arrives three people later. A master who has seen a billion scenes now sits next to you and, at each of 400 spots on the page, shows you which way the stroke should go. Not how dark the ink is, only the direction. You keep your own hand; you borrow the master's eye.

3. The Deep Semantic Injector

The Deep Semantic Injector (DSI) is the training-only part that does the pulling: a projector on F5 and a cosine loss against the teacher. Let's take its choices one at a time.

3.1 Where to inject?

The obvious candidates are the backbone maps S3, S4, S5 and AIFI's output F5. The paper tried them all on DEIM-L for 36 epochs, and measured how close each map already is to DINOv3's features with CKA [KORN19], a similarity score between two sets of features that ignores rotations and overall scale (0 unrelated, 1 equivalent).

Table. 2 Injection position (Tab. 3 of [LIAO25], DEIM-L, 36 epochs, COCO val2017 AP) beside the CKA of each map with the teacher (Tab. E). The paper uses CKA without citing a source; it is the measure of [KORN19].

• Pulling the backbone maps toward DINOv3 does nothing or hurts by 0.1. The paper's reading (App. F): S3 still holds low-level detail, is nearly unrelated to the teacher's space (0.05), and forcing it there disturbs its job.
• F5 alone is the only position that helps (+0.5), and it is already the closest map (0.59).
• Mixing backbone and F5 targets cancels the gain. The authors suspect the CNN maps and the attention map pull against each other (§4.2); the paper does not test that guess.
• One loss on F5 is also enough to reach everything below it: there is no stop on the student side, so the gradient flows back through AIFI, the S5 projection and the whole backbone.

3.2 Making the two grids meet

The teacher cuts its input into 16-pixel patches, so on a 640 image its grid is 40 × 40, twice F5's. The released code solves this before the teacher sees anything: it averages the image 2 × 2 down to 320 × 320, then patchifies.

The teacher's effective stride becomes 2 × 16 = 32, F5's own stride, so cell (i, j) of one grid sits on the same 32 × 32 patch of image as token (i, j) of the other. The paper describes an interpolation between the two grids (its Eq. 8); in the code that resize is a fallback that only fires when the input is not a multiple of 32.

3.3 The projector

F5 has 256 numbers per cell; DINOv3-B speaks in 768. A linear projector maps each cell $f_{ij}$ across:

That is 256 × 768 + 768 = 197,376 parameters for S/M/L (X has 384 channels: 295,680), and it is dropped at inference with the teacher. Tab. 4 compares projectors on DEIM-L: a 1 × 1 convolution 53.8, an MLP 54.2, linear 54.3. A 1 × 1 convolution and a per-cell linear layer are the same map on paper; the paper does not explain the 0.5 AP between them.

3.4 The loss: why cosine?

Let's devise the loss step-by-step. With $t_{ij}$ the teacher's token at cell (i, j) and H × W = 20 × 20, the paper's Eq. 9 asks every projected cell to point where the teacher points:

The code computes the mean of 1 − cos, which is the same loss plus 1 and has the same gradient. The teacher runs without gradients; only the student moves. Now the one property that matters, for any c > 0:

The cosine loss is blind to length. It asks for the direction and leaves the magnitude to the detector, which still needs its 256 numbers per cell for its own job; a squared error would also demand the teacher's magnitudes. Drag the student, then stretch it: the cosine loss does not move, the squared error does. The cosine push only turns the vector; the squared-error push drags its tip onto the teacher's.

length ×

1

same direction, twice as longfar off, shortReset

The measured version, on DEIM-M (90 + 12 epochs, Tab. 6): baseline 52.5, squared error 52.7 (+0.2), cosine 53.7 (+1.2).

3.5 Which teacher?

Same student (DEIM-L, 36 epochs), three ViT-B teachers (Tab. 5): MAE 53.9 [HE21], DINOv2 54.1 [OQUA23], DINOv3 54.3 [SIME25]. Of these three, the stronger teacher gives the bigger gain, and MAE barely helps (+0.1). DSI then goes into the total loss with a weight λ (the paper's Eq. 7):

${ℒ}_{det}$ is DEIM's detection loss, untouched. Everything now hangs on λ.

4. Gradient-guided Adaptive Modulation

4.1 Why a fixed λ fails

The paper trained DEIM-L eleven times with a fixed λ from 0.1 to 100 (50 + 8 epochs, Tab. 7). The best is λ = 20 at 55.1, against 54.7 for DEIM-L without the teacher (Tab. 1, same schedule). Too strong (λ = 100) falls back to 54.6.

The gradient table shows why a single value cannot be right for the whole run (Tab. C, L model):

• λ = 2 lifts AIFI's share from 0.51 % to only 1.12 %.
• λ = 20 lifts it to 2.01 % overall, but it overshoots early (2.55 % in epochs 1–12) and fades late (1.67 % by epochs 36–48). Its AP curve also climbs slower in early and mid training (Fig. 6).

So the right strength changes as the detector changes. GAM watches AIFI's share and moves λ to hold it.

fixed λ◀8 / 11 · λ = 20▶ NextGAM at epoch

0

▶best fixed λGAM late in trainingtoo strongReset

The shaded band is every λ that GAM itself used in the released L run: from 2 at the start to about 80. The best fixed value sits inside that band and still loses by 0.3 AP. The gain is in the schedule, not in the level.

4.2 Measuring the share

For each training step t, take the L1 norm of the gradient that each part c of the model receives (backbone, AIFI, CCFF, decoder), AIFI's fraction of the total, and its mean over the $T_e$ steps of epoch e (the paper's Eqs. 10–13):

Note the gradient is of the total loss. GAM does not regulate how big the alignment loss is; it regulates how much AIFI learns, from both losses together.

4.3 The update, and the far edge

At the end of each epoch, with a target share ρ and a tolerance δ (the paper's Eq. 14):

Read the targets carefully. Below the band, λ is scaled to aim at the upper edge ρ + δ, not at ρ. Above it, at the lower edge. Only part of AIFI's gradient comes from the alignment loss, so scaling λ moves the share by less than in proportion; aiming past the middle makes up for it (§3.4).

The authors released their training logs, so we can check it. The L model uses ρ = 2 % and δ = 0.1 % (Tab. A). The logged run starts at λ = 2 (Tab. A says 3; see Limitations). In epoch 0 the log prints a share of 1.53 %, below the band:

The log's own update is 2 → 2.755, a factor of 1.378. At epoch 29 the share dips to 1.69 % and Eq. 14 gives ×1.243; the log shows ×1.246.

NOTE: the released code does not use Eq. 14 exactly. It multiplies λ by the odds ratio p*(1 − p) / (p(1 − p*)), with p the measured share and p* the far edge, floors the factor at 0.1 and keeps the new λ within ×0.1 … ×10 of the old one. It also sets λ to a fixed value for the last epochs without strong augmentation. For shares around 2 % the odds ratio and the plain ratio agree to within 1 % (1.381 against 1.373 in epoch 0); for S, whose band sits near 11 %, they differ more. In the paper's λ units, the config value is divided by 5 (the config says its weight is 5 times the reported one).

Play the epochs below. λ climbs fast for six epochs, steps up whenever the share slips under 1.9 %, jumps again at epoch 29, when the learning rate starts to decay and mixup stops (both at epoch 29 in the released config), holds near 80, and drops to 3 for the final epochs. Then switch to "the rule by hand" and set the share yourself, for any size.

showtraining log (L)the rule by handepoch

0

▶

for the rule by hand

AIFI share r̄ (%)

1.53

ρ, δ of sizeSMLX

first epochthe dip at epoch 29EMA phaseby hand: L, 1.53 %by hand: S, 6 %Reset

4.4 Choosing ρ and δ

ρ and δ are per size (Tab. A):

Table. 3 GAM's settings per size (Tab. A of [LIAO25], λ in the paper's units).

For L, the pair (2 %, 0.1 %) was picked from a 24-run grid (Tab. D: ρ from 1 to 2.5 %, δ from 0 to 0.5 %). Every one of the 24 runs lands between 54.9 and 55.4, so every setting beats the baseline 54.7. Against the best fixed λ (55.1), 9 of the 24 are higher, 7 tie and 8 are lower: the band rule needs a good (ρ, δ) to win.

5. What It Costs to Train

The teacher is an 86 M-parameter ViT that runs a forward pass on every training batch. Tab. B prices it in RTX 3090 GPU-hours:

Table. 4 Training cost (Tab. B of [LIAO25]); the percentages recompute from the GPU-hours.

• The smaller the student, the larger the bill: an 86 M teacher next to a 10 M student costs S a sixth more time.
• Memory barely moves (+0.8 to +3.3 %): the teacher stores no gradients.
• Not in the table: the DINOv3 weights are a separate download (repository README).

6. Results

Protocol, as the paper states it: COCO val2017, latency on a T4 GPU with TensorRT FP16 (Tab. 1, App. A). Batch size and input size are not restated in v4's table, and its caption says the other detectors' numbers are "sourced from the official publications", without saying whether the YOLO times include NMS.

Table. 5 One network, three trainings (Tab. 1 of [LIAO25]). Params, GFLOPs and latency are shared by all three columns; D-FINE trains 120 / 120 / 72 / 72 epochs, DEIM and v4 120 / 90 / 50 / 50. These are the numbers of the October 2026 revision of the paper (arXiv v2).

v4 has no latency measurement of its own: the exported graph is DEIM's, so the latency is DEIM's by construction. The other real-time detectors at the L size, same table:

Table. 6 The L row of Tab. 1 of [LIAO25]. YOLO models train 500–600 epochs, v4-L 50.

• v4-L is +1.7 AP over YOLOv12-L, the best YOLO here, and 1.2 ms slower. The whole RT-DETR line before it sits at 53.1–53.4.

table◀4 / 5 · v4 paper: Table 1▶ Nextmarked version◀4 / 4 · v4▶ Next

Which families to show

YOLOshideshowother DETRshideshowRT-DETR familyhideshow

v1 paper's tablev2 paper's tablev3 paper's tablev4 paper's tablethe RT-DETR line, v1 markedthe RT-DETR line, v2 markedthe RT-DETR line, v3 markedthe RT-DETR line, v4 markedReset

The dotted verticals are the point of this post in one picture: D-FINE, DEIM and v4 share an x position at every size, because they are one network trained three ways.

NOTE: the RT-DETR rows of v4's table do not all match v1's own paper. RT-DETR-R34 is 6.91 ms here, against 161 FPS (6.21 ms) in [ZHAO24] and 6.3 ms in [WANG24]; R101 is 13.88 ms here against 74 FPS (13.51 ms). R50 agrees (9.29 against 108 FPS, 9.26 ms). Possibly a re-measurement; the paper does not say. I quote each number with its own table and never mix them in one plot.

Is it a plug-in? Tab. 2 adds DSI, then GAM, to three detectors (L size, 36 epochs):

Table. 7 DSI and GAM on three detectors (Tab. 2 of [LIAO25], COCO val2017 AP, 36 epochs).

• It helps all three, by +0.3 to +0.5.
• DSI alone gives only +0.1 to +0.2; most of the gain needs GAM. The paper does not say which fixed λ the "+ DSI" column used.

7. What Ships and What Stays Behind

Part by part, for RT-DETRv4:

Table. 8 RT-DETRv4, part by part, from §3, §5 and Tab. 1 of [LIAO25]. Training pays +6 to +16 % GPU-hours (Table. 4).

The series ledger, with v4's row:

Table. 9 The series ledger after part 4. Rows v1–v3 from RT-DETR — DETRs Beat YOLOs, Encoder by Encoder, RT-DETRv2 — A Bag of Freebies for the Decoder and RT-DETRv3 — More Positives Without Paying at Inference; row v4 from Tab. 1 of [LIAO25].

version◀4 / 5 · v4▶ Nextstage◀4 / 8 · AIFI on S5▶ Next▶modeinferencetrainingmodel sizeR18 / SR34 / MR50 / LR101 / Xwhole pipelinev1 at inferencev1 in trainingv2 at inferencev3 in trainingv3 at inferencev4 in trainingv4 at inferenceWorld in trainingWorld at inferenceReset

8. Limitations

The paper's own claims are above. What follows is my reading of where they reach past what was tested.

• "Without additional tuning overhead." §3.4 says GAM gives stable convergence "without additional tuning overhead". Yet ρ runs from 2 to 11 % across sizes, the starting λ and the last-epochs λ are set per size, and L's pair came from a 24-run grid (Tab. D). GAM is robust once tuned; it is not tuning-free.
• Small margins, single runs. Most ablation gaps are 0.1–0.5 AP and no seeds or spreads are reported. GAM's edge over the best fixed λ is 0.3 AP (55.4 against 55.1).
• Scope. COCO only, and only DETR-style students (RT-DETRv2, D-FINE, DEIM). §5 says the framework also suits CNN detectors, other foundation models such as CLIP, and can "even benefit from arbitrarily large models". Only three ViT-B teachers were tested (Tab. 5); no CNN detector, no CLIP, no larger teacher.
• Theory as motivation. The information-bottleneck framing (§3.1) is not measured. The evidence for "inject deep" is Tab. 3 and the CKA numbers of Tab. E.
• Code and paper drift. The odds-ratio rule and its clamps (§4.3 NOTE); the starting λ for L and X is 3 and 2 in Tab. A, 3 and 4 in the configs, and 2 and 2 in the released logs; the 1 × 1 convolution and the linear projector are the same map but score 0.5 AP apart (Tab. 4).

9. Conclusion

• v4 lets a frozen DINOv3 coach F5, cell by cell and by direction only, on top of DEIM-trained D-FINE.
• GAM keeps the coaching steady by watching AIFI's share of the gradient; λ climbs about 40-fold (2 → 80 for L), a schedule no fixed value reproduces.
• The coach is thrown away before deployment. The bill is +6 to +16 % GPU-hours, for +0.5 to +1.0 AP over the same network trained without it.

Venue: the arXiv record (2510.25257, v2 of 8 October 2026) carries none. Only the authors' repository README says ECCV 2026, so I cite it as an arXiv preprint.

10. What Came Next

v4 used DINOv3 as a training-only teacher and threw it away. The last post of this series, RT-DETR-World — Naming Any Object in Real Time, comes from a different group and does not cite RT-DETR v1–v4. It keeps a frozen DINOv3 as the deployed backbone, replaces the 80 fixed classes with names you type, and uses a language model's embeddings only as precomputed training targets. My observation, not the paper's: the foundation model moves from coach to body, and the language model becomes the coach. Stay tuned.

11. References

1. [LIAO25] Liao, Zhao, Shan, Yan, Liu, Lu, Ji, Chen. RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models. arXiv 2510.25257 (v2, October 2026); ECCV 2026 per the authors' repository.
2. [PENG24] Peng, Li, Wu, Zhang, Sun, Wu. D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinement. ICLR 2025, arXiv 2410.13842.
3. [HUAN24] Huang, Lu, Cun, Yu, Zhou, Shen. DEIM: DETR with Improved Matching for Fast Convergence. CVPR 2025, arXiv 2412.04234.
4. [ZHAO24] Zhao, Lv, Xu, Wei, Wang, Dang, Liu, Chen. DETRs Beat YOLOs on Real-time Object Detection. CVPR 2024, arXiv 2304.08069.
5. [LV24] Lv, Zhao, Chang, Huang, Wang, Liu. RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer. 2024, arXiv 2407.17140.
6. [WANG24] Wang, Xia, Lv, Shi. RT-DETRv3: Real-time End-to-End Object Detection with Hierarchical Dense Positive Supervision. WACV 2025, arXiv 2409.08475.
7. [SIME25] Siméoni et al. DINOv3. 2025, arXiv 2508.10104.
8. [OQUA23] Oquab et al. DINOv2: Learning Robust Visual Features without Supervision. 2023, arXiv 2304.07193.
9. [HE21] He, Chen, Xie, Li, Dollár, Girshick. Masked Autoencoders Are Scalable Vision Learners. 2021, arXiv 2111.06377.
10. [KORN19] Kornblith, Norouzi, Lee, Hinton. Similarity of Neural Network Representations Revisited. ICML 2019, arXiv 1905.00414.
11. [TISH00] Tishby, Pereira, Bialek. The Information Bottleneck Method. 2000, arXiv physics/0004057.
12. RT-DETRv4 code, configs and training logs: github.com/RT-DETRs/RT-DETRv4 (the GAM figure replays logs/ for the L model).