RT-DETR-World — Naming Any Object in Real Time

Every detector so far in this series ends in 80 fixed outputs, one per COCO class. Type "wheelchair" and none of them can answer: there is no 81st output to read. An open-vocabulary detector fixes that by making the class list something you type at test time. RT-DETR-World [ZHAN26] (arXiv, 7 Oct 2026) is a real-time one, and the whole post hangs on three steps:

• Look: a frozen DINOv3 [SIME25] turns the image into features, and a small decoder turns them into 300 object queries, each a box and a vector.
• Name: every name you type becomes a vector once, through a small text encoder (MiniLM [WANG20a]); each query is scored against each name.
• Teach: while training, an LLM's sense of which descriptions mean similar things shapes the loss. It arrives as numbers computed before training starts, and none of it ships.

First, whose model is this? A different group. The authors are from Tianjin University, Shenzhen University of Advanced Technology, Shanghai Jiao Tong University and Shanghai AI Laboratory; none of them is an author of RT-DETR RT-DETR — DETRs Beat YOLOs, Encoder by Encoder, v2, v3 or RT-DETRv4 [LIAO25]. The paper does not cite RT-DETR v1, v2, v3 or v4, and its reference list has no DETR [CARI20] either. The name is the only stated link. So every comparison with parts 1–4 below is my observation, not the paper's claim. The one I find most interesting: v4 used DINOv3 as a teacher that is thrown away before deployment; World ships DINOv3 as its frozen backbone and makes a language model the teacher that is thrown away.

version◀5 / 5 · World▶ Nextstage◀8 / 8 · boxes and classes▶ Next▶modeinferencetrainingmodel sizeR18 / SR34 / MR50 / LR101 / Xwhole pipelinev1 at inferencev1 in trainingv2 at inferencev3 in trainingv3 at inferencev4 in trainingv4 at inferenceWorld in trainingWorld at inferenceReset

Brush up before you start ;)

• Contrastive image–text training, InfoNCE in both directions and the temperature τ: CLIP — Learning Pictures From the Words Next to Them. I use them here without re-deriving them.
• DINOv3 as a training-only teacher, and the "teach in training, remove at inference" move: RT-DETRv4 — Borrowing a Foundation Model's Eyes.
• Object queries, one-to-one matching and the set loss: DETR — Detection as Set Prediction, No Anchors and No NMS. Query groups in training: RT-DETRv3 — More Positives Without Paying at Inference.

1. Open Vocabulary: the Classifier Becomes a List of Names

Think of a cloakroom. A closed-set detector has 80 numbered hooks bolted to the wall; a coat with no hook gets turned away. An open-vocabulary detector has a clipboard instead: write a new name on it and the coat has somewhere to go. Nothing on the wall changed.

In a closed-set detector the last layer is a weight matrix with one row per training class. In an open-vocabulary one, the rows are the text vectors of whatever names the user types. Let ${𝐪}_j$ be the vector of object query $j$ and ${𝐞}_k$ the vector of the $k$-th name. The score is

• $K$ is however many names you typed: 1,203 for LVIS [GUPT19], or 2 for "wheelchair, crutch".
• The name vectors come from MiniLMv2-L6-H768 (6 layers, width 768). They are computed once per vocabulary and cached, and the paper's speed numbers are measured that way (Table 2 notes).
• I wrote "sim" on purpose. The paper does not give the exact score: cosine or dot product, a learned scale or bias, focal or another classification loss are all unstated. The detection loss is described only as Hungarian-matched classification and box regression (§3.4).
• The same similarity also picks the starting queries ("Language-Guided Query Selection", §3.1). How exactly (top-k of which score) is not stated either.

Let's see what "a list you can extend" means with a toy:

query direction (degrees)

322

namessix namesadd “wheelchair”six namesadd wheelchairquery near the busReset

NOTE: there is no temperature control on that figure, on purpose. The only temperature the paper states, τ = 0.07, belongs to the training-only alignment loss of Section 5, not to the detection score.

At test time you type plain category names; the descriptions of Section 4 are training supervision only (§1).

2. What Ships: the Deployed Detector

Here is the detector as the paper states it (Fig. 3, §3.1, §3.4), part by part:

• Backbone: a frozen DINOv3 ViT with 16-pixel patches, in three sizes: ViT-S/16 (World-S), ViT-S+/16 (World-S+) and ViT-B/16 (World-B). It is not fine-tuned at all.
• A lightweight multi-scale projector: two layers, 256-dimensional features. How one stride-16 ViT grid becomes several scales, and how many scales there are, is not stated.
• Language-Guided Query Selection: initial queries picked by vision–text similarity. Rule not stated.
• A DETR decoder with three layers and 300 queries at inference. Whether it uses deformable attention, denoising queries or box refinement is not stated.
• Text encoder: MiniLMv2-L6-H768, at most 256 tokens. It is trained (learning rate × 0.1), not frozen, and it ships.

Where the paper is silent, the architecture figure says so; I am not filling those parts in.

Table. 1 Parameters and speed of the three deployed models (paper Table 2). FPS on an NVIDIA T4 with TensorRT, following OV-DEIM's protocol [WANG26a]; the input size of the speed runs is not stated.

Two numbers to hold on to:

• The MiniLM is 66.6 M parameters, more than the whole visual side of World-S. It is cheap only because it runs once per vocabulary, not once per image.
• 103–187 M in total is larger than any RT-DETR v1–v4 model, and the FPS cannot be put next to their COCO latencies: different task, different metric, and evaluation at shorter side 800 (longer side up to 1,333), not 640 × 640.

3. Where This Sits in the Series (My Reading)

Since the paper says nothing about RT-DETR, here is the comparison I draw, side by side:

Table. 2 v4 against World. The v4 column is from the v4 paper (see RT-DETRv4 — Borrowing a Foundation Model's Eyes); the World column from §3. The pairing is mine; World does not compare itself with v4.

The shared pattern: a big model teaches during training and stays behind at deployment. World changes what teaches: language instead of pixels.

4. The Data: Three Levels of Text

A class name is a thin signal. "Sofa" tells you nothing about "two-seat" or "leather". The paper's bet (Fig. 1, §1): if training also shows the model descriptions full of parts, counts and materials (Fig. 1 has "two wooden wheels leaning against a wall"), cues such as chair + wheel, two-seat + sofa or wooden + spoon may help it recognise names it never saw: "wheelchair", "two-seater sofa", "wooden spoon".

So the authors rebuild LLMDet's GroundingCap-1M [FU25] into GroundingCapv2, with three levels of text per sample (§2.1):

• category names: the interface you use at test time;
• object descriptions: attributes, parts, quantities, actions, poses and states of one box;
• an image description: relations and scene context.

Table. 3 GroundingCapv2 by source (paper Table 1). The sample and region columns add up to the total row; the total coverages are the region-weighted means of the rows (my arithmetic).

Two language models appear in this paper, with two different jobs. Keep them apart:

• Qwen3-VL writes and checks the data. An 8B proposer writes an instance description from the image, the box and the category; an independent 8B verifier checks it; unsure cases go to a 32B adjudicator (§2.2). For COCO and V3Det, 85.5 % and 87.6 % of regions got an accepted description; the rest fall back to the category name.
• LLM2CLIP-Llama-3.1-8B [HUAN24a] is the teacher. It turns every object description and image description into a 1,152-dimensional vector, once, offline. The LLM is never loaded while the detector trains or runs (§3.2).

5. Dual-Path Description Alignment

The paper's worry (§3.2): a compact detector may struggle to absorb rich text through one thin path. So it uses two, and they do different jobs:

• Path 1 trains what ships. It runs through MiniLM and the queries, the two things you keep. The paper calls it "deployment-consistent" (§3.2).
• Path 2 borrows a sharper target. It pulls the queries toward the teacher's 1,152-d vectors. Its projection heads are thrown away.

Let's write them. For sample $b$, region $i$ has description $d_{bi}$, and ${𝐪}_{b,\pi (i)}$ is the final-layer query that the matching assigned to it. Path 1 mean-pools MiniLM's token features of the description and projects both sides into one shared space (paper Eq. 1):

• $E_m$ is MiniLM; $P_m^t$, $P_m^v$ are learned projections. The gradient lands in MiniLM and in the queries.
• The width of the shared space is not stated.

Path 2 maps the matched query into the teacher's space, and also builds one global vector for the whole image (paper Eq. 2):

• ${𝐯}^{tch}$ is pulled toward the teacher's vector of the object description: what is this thing.
• ${𝐠}_b$ is pulled toward the teacher's vector of the image description: what is going on here. MAP is a mask-aware average over the valid (unpadded) positions of each projected feature level $F_b^l$, and the levels are concatenated.
• $P_o$, $P_g$ and the global branch exist only in training.

Switch the architecture figure to training: the teacher's precomputed vectors and the 12 extra query groups appear as dashed, training-only blocks. The four projections and the global branch are not drawn; the readout lists them:

version◀5 / 5 · World▶ Nextstage◀7 / 8 · decoder layers▶ Next▶modeinferencetrainingmodel sizeR18 / SR34 / MR50 / LR101 / Xwhole pipelinev1 at inferencev1 in trainingv2 at inferencev3 in trainingv3 at inferencev4 in trainingv4 at inferenceWorld in trainingWorld at inferenceReset

The paper also trains with 13 groups of 300 queries and keeps one group of 300 at inference (§3.4). That reminds me of the query groups of RT-DETRv3 — More Positives Without Paying at Inference (my link, not the paper's). Several query groups through one decoder is Group DETR's idea [CHEN22], which the World paper neither names nor cites; how World's groups are matched or masked is not stated.

6. Relation-Aware Negative Relaxation

All three alignments are contrastive: in a batch, each visual vector must pick its own description out of the others. Here is the problem. Picture a room full of bus photos. One description says "a red double-decker bus", another "a parked city bus". Standard contrastive training shouts "wrong!" at the city bus exactly as loudly as at "a wooden spoon". But the two buses share the very factors the model is supposed to learn. Relation-Aware Negative Relaxation (RNR) lowers the voice for related wrong answers, by how related the teacher thinks they are (§3.3).

Let's devise the equations step by step. A batch has $N$ triplets $({𝐯}_n,{𝐭}_n,{𝐞}_n)$: a visual vector, its paired text vector, and the frozen teacher vector of that text. First, the logits in both directions, cosine over a temperature, as in CLIP — Learning Pictures From the Words Next to Them (paper Eq. 3):

with $\tau =0.07$. Next, the change: a weight for every pair, read from the teacher (paper Eq. 4). $P_n$ is the set of true matches of anchor $n$, usually just $\{n\}$:

• $r_{nm}$ is how related the teacher finds the two descriptions, clipped at 0: unrelated pairs get no discount.
• Negative weights lie in $[1-\alpha ,1]$. The paper uses $\alpha =0.25$, so a negative is never discounted by more than a quarter.

The weights go into the denominator only (stated just after Eq. 4):

and the objective is the mean over anchors and both directions, $\frac{1}{2N}{\sum }_n({\ell }_n^{v2t}+{\ell }_n^{t2v})$, with the same weights in both directions.

• Put $\alpha =0$: every weight is 1 and this is the ordinary symmetric InfoNCE of CLIP — Learning Pictures From the Words Next to Them. RNR is a one-knob change.
• The true match keeps weight 1 and stays in the numerator. A related negative is pushed less, never pulled: in the paper's words it is "relaxed but never treated as a positive" (§3.3). That is the difference from soft-target methods, which blur the positive itself.
• Why the teacher's similarities, not MiniLM's? The teacher is frozen, so the weights do not drift, and the paper argues it judges paraphrase better (§3.3).

How much less is a negative pushed? Differentiate the loss with respect to one negative logit. This step is mine; it follows from the formula and is not printed in the paper:

The push on a negative scales with its weight. Numbers to convince, on a toy anchor with τ = 0.07: the true description at cosine 0.6, the city bus at 0.5 with teacher relation r = 0.8, a spoon at 0.1 and a cat at 0.0 (both r = 0).

• α = 0: push on the city bus 0.193, loss 0.216.
• α = 0.25 (the paper's): push 0.161, loss 0.176.
• α = 0.5: push 0.126, loss 0.135.
• The spoon and the cat barely register either way (about 0.0006 and 0.0002): at τ = 0.07 they are already far behind.

α, how much to relax (paper: 0.25)

0.25

teacher relation r of the city bus

0.8

temperature τ (paper: 0.07)

0.07

standard contrastive (α = 0)the paper's α = 0.25too much (α = 0.5)Reset

RNR adds no term of its own; it only reweights the three alignment losses (paper Eq. 5):

• $L_M^{obj}$: projected queries ${𝐯}^m$ against projected MiniLM descriptions $𝐦$. Here the teacher supplies only the weights.
• $L_L^{obj}$: ${𝐯}^{tch}$ against the teacher's object vectors, weights from those same vectors.
• $L_L^{img}$: $𝐠$ against the teacher's image vectors, weights from those.

7. The Total Loss

• $({\lambda }_M,{\lambda }_o,{\lambda }_g)=(0.25,0.1,0.1)$, τ = 0.07, α = 0.25. The three description losses ramp up linearly over the first 2,000 iterations (§3.4).
• 150k iterations on 4 × RTX 4090, batch 48 (12 per GPU), AdamW at 3 × 10⁻⁴, weight decay 10⁻⁴, MiniLM at 0.1 × the learning rate, drops at 120k and 140k.
• Not stated: the GPU-hours, the cost of writing GroundingCapv2 with Qwen3-VL, and the cost of precomputing the teacher vectors.

8. Results

8.1 Two kinds of AP

Everything here is zero-shot on LVIS minival: trained only on GroundingCapv2, then given the 1,203 LVIS names (337 rare, 461 common, 405 frequent), with no LVIS labels in training (§4.1). Two metrics:

• Standard AP keeps at most 300 detections per image, across all classes. With 1,203 classes, frequent ones can crowd rare ones out of those 300.
• Fixed AP [DAVE21] gives each class its own budget instead of a per-image cap; here the top 1,000 predictions over all query–class pairs. It measures ranking within a class, not the fight for slots.

The paper's "best real-time" claim is in Fixed AP. Let's see whether it holds in both:

Table. 4 LVIS minival, zero-shot (paper Table 2, real-time rows and the two strongest heavy models). * FPS includes NMS. The table does not mark RT-DETR-World as using NMS. AP_r is on rare classes.

• In Fixed AP the claim holds. World-S+ is 36.6 against 35.9 for both YOLOEv8-L and OV-DEIM-L: +0.7. World-B is 40.0: +4.1.
• In standard AP it holds only for B. World-S (31.4) and World-S+ (32.6) are below OV-DEIM-L (33.7) and YOLOEv8-L (33.3). Only World-B (36.6) leads. This is my reading of the same table; the paper reports both columns but frames the claim in Fixed AP.
• Speed is the price. OV-DEIM-L gives 35.9 Fixed AP at 91 FPS; World-S+ buys +0.7 at 58 FPS, World-B +4.1 at 32 FPS, as the paper itself concedes.
• Heavy models still lead. World-B is 1.4 below MM-Grounding-DINO-T and 4.7 below LLMDet-T in Fixed AP. Both are Grounding DINO-style models [LIU23] (LLMDet starts from MM-Grounding-DINO, §4.1 of [FU25]) with a Swin-T backbone and a BERT text encoder, and the table gives no FPS for them.
• Different training data. World trains on GroundingCapv2; YOLO-World, YOLOE and OV-DEIM on Objects365 + GoldG. The comparison mixes data and method.

show◀1 / 6 · Fixed AP▶ NextFixed APstandard APrare classesspeed (FPS)ablation APablation rareReset

8.2 Beyond LVIS

• ODinW [LI22a], mean AP over the usual 13-dataset subset and over all 35 datasets, from many domains (Table 3): World-S 40.1 / 18.7, World-B 43.8 / 20.2, against OV-DEIM-L 41.6 / 19.2. B leads by +2.2 / +1.0.
• COCO-O [MAO23], the 80 COCO classes as sketches, cartoons, paintings, tattoos, handmade objects and in bad weather (Table 4): World-B 46.2 AP against OV-DEIM-L 43.3. GroundingCapv2 contains COCO training images, so the paper itself treats this as a distribution-shift test, not an unseen-class test.

8.3 What each piece buys

All ablations are on World-S, Fixed AP (§4.3). Table 5 of the paper switches the supervision on piece by piece:

Table. 5 Supervision ablation on World-S, LVIS minival Fixed AP (paper Table 5). The B rows learn classes from the sources' own categories or region phrases; the C rows from GroundingCapv2's category names through MiniLM.

• The teacher alone, on the old data: B0 → B1, +5.0 AP. This is the only row pair that isolates the method from the new data.
• The MiniLM path on top of the teacher: C0 → C4, +2.1 AP, and AP_r 25.7 → 32.0.
• Rare classes move most once the new data is in. In every step between C rows AP_r gains more than AP_c and AP_f; in B0 → B1 the common classes edge them (+5.4 against +5.3, paper Table 5). My reading: rare classes are the ones least likely to be in the training vocabulary by name, so that is where descriptions should help.

And the loss itself (paper Table 6, same setting): hard contrastive 33.6, SoftCLIP-style soft targets 33.8, SRCL-style reweighting 33.9, RNR 35.2. That is +1.6 AP and +4.4 AP_r over hard contrastive.

NOTE: the method is sensitive. In the paper's Table 7, α = 0.5 gives 33.4, below α = 0 (33.6): too much relaxation is worse than none. The teacher weight λ_aux = λ_o = λ_g = 0.05 gives 31.6, below switching the teacher off (32.2). No seeds or error bars are reported, and several differences are under 1 AP.

9. What Ships and What Stays Behind

Table. 6 RT-DETR-World, part by part, from §3.1–§3.4 and Table 2 of the paper.

So: the new training parts add nothing at inference; the model still runs a text encoder once per vocabulary. The series ledger, with this row added:

Table. 7 The series ledger. Each row is from that post and its paper.

10. Limitations

Stated by the paper:

• World-B is still 1.4 Fixed AP below MM-Grounding-DINO-T and 4.7 below LLMDet-T (§4.2).
• OV-DEIM-L is faster at similar Fixed AP; World trades latency for accuracy (§4.2).
• Dense scenes still give low-confidence and overlapping boxes; fine-grained discrimination and confidence calibration over a large vocabulary stay hard (Appendix C). There is no dedicated limitations section.

My observations, from reading the paper against its own tables:

• "Best real-time" is a Fixed AP statement. In standard AP, S and S+ trail OV-DEIM-L and YOLOEv8-L.
• Training data differ from the baselines'. Only B0 → B1 isolates the method, and only for the teacher.
• Only image overlap with the test sets is removed (§4.2). Overlap of names, for example how many LVIS names also appear among V3Det's categories, is not analysed.
• The decoder, query selection, projector, detection score and the speed runs' input size are not described. The code "will be released" (abstract); until then the model cannot be rebuilt from the paper.
• No seeds or variance, and the sensitivity table is not monotone.
• The PDF header says "Published as a conference paper at ICLR 2027". I could not confirm that, so I cite it as an arXiv preprint.

11. Conclusion: Five Posts, One Move

RT-DETR-World swaps the 80 fixed outputs for names you type, keeps DINOv3 frozen as its backbone, and borrows an LLM's sense of which descriptions are related, only through targets computed before training. That buys the best Fixed AP among the real-time detectors it compares with, but not the best standard AP at every size.

Looking back at the whole series:

• v1 made a DETR real-time, and already trained with parts it throws away.
• v2 and v3 kept v1's inference graph (v2's two decoder dials are optional) and moved the gains into training: a recipe, then more positives. v4 moved to DEIM's network and kept that graph unchanged, with a foundation model as a training-only coach.
• World, from a different group and on a new task, ships a new network, and repeats the move with language: the coach is an LLM that is never even loaded in training.

The lesson I take: once the inference graph is fast enough, the cheapest accuracy is the kind you pay for only while training. Stay tuned for the code ;)

12. References

1. [ZHAN26] Zhang et al., RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection, arXiv 2610.09502 (preprint; the ICLR 2027 header is unconfirmed).
2. [LIAO25] Liao et al., RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models.
3. [SIME25] Siméoni et al., DINOv3.
4. [RADF21] Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP).
5. [CARI20] Carion et al., End-to-End Object Detection with Transformers (DETR).
6. [WANG20a] Wang et al., MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers.
7. [HUAN24a] Huang et al., LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representation.
8. [FU25] Fu et al., LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models.
9. [CHEN24] Cheng et al., YOLO-World: Real-Time Open-Vocabulary Object Detection.
10. [WANG25a] Wang et al., YOLOE: Real-Time Seeing Anything.
11. [WANG26a] Wang et al., OV-DEIM: Real-time DETR-Style Open-Vocabulary Object Detection with GridSynthetic Augmentation.
12. [LIU23] Liu et al., Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection.
13. [ZHAO24a] Zhao et al., An Open and Comprehensive Pipeline for Unified Object Grounding and Detection (MM-Grounding-DINO).
14. [GUPT19] Gupta et al., LVIS: A Dataset for Large Vocabulary Instance Segmentation.
15. [DAVE21] Dave et al., Evaluating Large-Vocabulary Object Detectors: The Devil is in the Details (Fixed AP).
16. [LI22a] Li et al., ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (ODinW).
17. [MAO23] Mao et al., COCO-O: A Benchmark for Object Detectors under Natural Distribution Shifts.
18. [CHEN22] Chen et al., Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment.