Vision Transformers Today — May 2022 to October 2026, Year by Year

By the end of 2021 the pieces were on the table: the transformer block, the patch, the window, a way to pre-train from pixels (Masked Autoencoders — Hide 75 %, Encode the Rest) and a way to pre-train from captions (CLIP — Learning Pictures From the Words Next to Them). This last post is what people built with them from May 2022 to October 2026, in the order they built it. One entry per model: why it exists, how it changes the computation, what it measured, and what it changes about an earlier post.

Nothing is re-derived here. Attention and its $n^2$: Query, Key, Value — Dot-Product Attention from Scratch; the block: Attention Is All You Need — The Transformer, Block by Block; patches: An Image Is Worth 16×16 Words — ViT, Patch by Patch; windows: Swin Transformer — Windows, Shifts and a Pyramid for Dense Prediction; object queries: DETR — Detection as Set Prediction, No Anchors and No NMS; distillation: DeiT — Training a Vision Transformer on ImageNet Alone; the 2021 hybrids: Convolutions Strike Back — Was It the Attention or the Recipe?. Left out by choice: the language decoders stacked on these encoders, video generation systems, Gaussian splatting.

1. 2022: The Kernel and the Generator

FlashAttention (May 2022) [DAO22]

• The $N\times N$ score matrix goes out to the GPU's big slow memory (HBM) and comes back, several times per pass; that traffic, not the arithmetic, is the bottleneck.
• Tile $Q,K,V$ into blocks that fit the small fast memory (SRAM), run the softmax online, never write the matrix out, recompute it in the backward pass. Exact attention, no approximation. The trick that keeps tiling exact is three lines:

• $s$ are the scores of the new block, $m$ and $\ell$ the running maximum and sum, $o$ the unnormalised output so far; $o/\ell$ after the last block is the exact output. Each block rescales what was already summed by $e^{m-m^{\prime }}$, so the softmax is the true one although no row was ever seen whole.
• Theorem 2: $\Theta (Nd+N^2)$ HBM accesses for standard attention against $\Theta (N^2{d}^2/M)$, $M$ the SRAM size. On GPT-2 medium (Figure 2): HBM traffic 40.3 → 4.4 GB, runtime 41.7 → 7.3 ms, while the GFLOPs rose from 66.6 to 75.2.
• What it changes: the $n^2$ of Query, Key, Value — Dot-Product Attention from Scratch became cheap in memory, not in FLOPs. Time is still $O(N^2)$.

DiT (Dec 2022) [PEEB22]

• Diffusion models used U-Nets. Is that inductive bias needed?
• Patchify the latent (a 256² image is a 32 × 32 × 4 latent; $p=2$ gives 256 tokens) and run a plain ViT whose only new piece is the conditioning, adaLN-Zero:

• $c$ is the timestep-plus-class embedding; $\gamma ,\beta$ are the LayerNorm scale and shift regressed from it; $\alpha$ gates the residual branch and starts at zero, so every block starts as the identity. Best of four options (Figure 5).
• Table 2: DiT-XL/2 FID 2.27 on ImageNet 256² (ADM 10.94, LDM-4-G 3.60) at 118.6 Gflops against 742 for ADM-U; FID tracks Gflops, not parameters (Figure 8).
• What it changes: the block of Attention Is All You Need — The Transformer, Block by Block on the patches of An Image Is Worth 16×16 Words — ViT, Patch by Patch, of a latent. Stable Diffusion 3 [ESSE24] builds on it below.

2. 2023: Scale, Segmentation and Features That Transfer

ViT-22B (Feb 2023) [DEHG23]

• Can a ViT be scaled like a language model?
• 48 layers, width 6144, 48 heads, patch 14 (Table 1); parallel attention and MLP; QK LayerNorm, since attention logits diverged near 8 B parameters without it (Figure 2); no QKV biases; about 4 B JFT images at batch 65,536 on 1024 TPU v4.
• 21.743 B parameters; ImageNet linear probe 89.5 %; LiT zero-shot 85.9 %. Check the count: $12D^2=12\times {6144}^2=452,984,832$ per block, times 48 $=21,743,271,936$.
• What it changes: the data-against-inductive-bias story of An Image Is Worth 16×16 Words — ViT, Patch by Patch taken to its end. The largest reported ViT, never released.

SigLIP (Mar 2023) [ZHAI23]

• The softmax of CLIP — Learning Pictures From the Words Next to Them needs the whole $N\times N$ normalisation and huge batches.
• Score every pair alone with a sigmoid, with a learned temperature and bias:

• $z_{ij}=+1$ on the diagonal, $-1$ elsewhere; $t^{\prime }$ starts at $\log 10$ and $b=-10$ offsets a start where almost every pair is a negative (§3.2). No row is normalised, so the loss is chunked across devices with no all-gather of the matrix (§3.3).
• Table 1: sigmoid beats softmax clearly below a 16 k batch and "performance saturates at 32 k batch size"; a locked-image SigLiT g/14 reaches 84.5 % ImageNet zero-shot in two days on four TPUv4. The loss that replaced CLIP's in most new encoders.

SAM (Apr 2023) [KIRI23]

• A foundation model for segmentation needs a promptable task and a dataset that did not exist.
• An MAE-pre-trained ViT-H/16 encoder (636 M parameters, 1024² input, windowed attention with four global blocks), a prompt encoder for points, boxes and masks, and a light decoder that emits three masks plus IoU scores, because a click on a shirt may mean the shirt or the person. A data engine produced SA-1B: 11 M images, 1.1 B masks, 99.1 % automatic.
• Beats RITM from a single point on 16 of 23 datasets (Figure 9); COCO instance AP 46.5 with ViTDet-H boxes against ViTDet-H's own 46.6 (Table 5).
• What it changes: the encoder is Masked Autoencoders — Hide 75 %, Encode the Rest, fine-tuned end to end; the decoder's output tokens are the queries of DETR — Detection as Set Prediction, No Anchors and No NMS. Say "promptable", not "open-vocabulary": finding all buses needs a detector in front.

DINOv2 (Apr 2023) [OQUA23]

• Self-supervised features good enough to use frozen.
• DINO's image-level self-distillation plus the iBOT masked-patch loss that predicts a teacher's latents; LVD-142M curated by retrieval from 1.2 B web images; a 1.1 B ViT-g/14 distilled into S/B/L; FlashAttention and sequence packing, "around 2× faster using only 1/3 of the memory" of iBOT.
• Table 4: ImageNet linear 86.5 % frozen, above OpenCLIP ViT-G/14's 86.2, which saw text; ADE20K 49.0 mIoU with a linear layer on frozen patches (Table 10); NYUd RMSE 0.344 (Table 11).
• What it changes: MAE's masked patches with latents, not pixels, as the target. Not contrastive: no text, no negatives.

Hiera (Jun 2023) [RYAL23]

• Hierarchical ViTs in the style of Swin Transformer — Windows, Shifts and a Pyramid for Dense Prediction carry convolutions, shifted windows and relative-position biases that are slow at run time; Hiera strips them out and lets MAE pre-training supply what they supplied.
• "Strip out all the bells-and-whistles ... without losing accuracy" (abstract), faster on images and video. SAM 2's encoder a year later.

NaViT (Jul 2023) [NAVIT23]

• Squashing every image to a square wastes pixels, so NaViT packs the tokens of several images of any resolution into one sequence, with masked attention and factorised positions: sequence packing.
• Better training efficiency, and resolution as an inference-time cost knob (abstract). The tokens of An Image Is Worth 16×16 Words — ViT, Patch by Patch are a set; SigLIP 2's NaFlex ships this in 2025.

FlashAttention 2, 3, 4 (Jul 2023 to Mar 2026) [DAO23] [SHAH24] [ZADO26]

• Three more generations of the one idea, move less data, each rewritten for the next GPU; none changes an output. From the abstracts: FlashAttention-2 (Jul 2023) [DAO23] about 2× over the first, 50–73 % of A100 peak; FlashAttention-3 (Jul 2024) [SHAH24] 740 TFLOPs/s on the H100, 75 % utilisation in FP16, about 1.2 PFLOPs/s in FP8; FlashAttention-4 (Mar 2026) [ZADO26] 1613 TFLOPs/s, 71 % of a B200, where the softmax is now the bottleneck because Blackwell doubled the tensor cores but not the exponential units, "asymmetric hardware scaling".

Registers (Sep 2023) [DARC23]

• ViT attention maps show high-norm "artifact" tokens in dull background patches, which the network repurposes as scratch memory; add a few learnable register tokens to the input and discard them at the output.
• Artifacts gone, dense prediction and object discovery better, attention maps smooth (abstract). They are extra inputs, not outputs; ViT-5 ships four of them.

Mamba (Dec 2023) [GU23]

• Attention's cost grows with $N^2$; recurrent models compress the past into a fixed state, but earlier linear-time state-space models could not select what to keep.
• Make the state-space parameters depend on the input and run the recurrence as a fused scan in SRAM with recomputation, FlashAttention's IO argument again. For one channel, with state size $n$:

• $A\in {ℝ}^{n\times n}$ is a learned diagonal state matrix; $B_t,C_t$ and a step size ${\Delta }_t$ are regressed from the token, and ${\overline{A}}_t,{\overline{B}}_t$ are $A$ and $B_t$ scaled by that step. Token-dependent is what "selective" means, and what breaks the convolution shortcut of earlier models.
• ${\Delta }_t$ is the gate: "large $\Delta$ resets the state and focuses on the current input, while small $\Delta$ persists the state and ignores the current input".
• Attention keeps every past token on the table, $O(N^{2}d)$; the scan folds the past into $n$ numbers per channel, $O(NnD)$. "Attention ... does not compress context at all." 5× a transformer's inference throughput on language (abstract). No vision yet.

3. 2024: The State-Space Year

Vision Mamba, Vim (Jan 2024) [ZHU24]

• ViT patch tokens, a class token in the middle, each block scanning forward and backward because images have no causal order.
• The paper's cost formulas (its Eq. 5 and 6), $M$ tokens, width $D$, expansion to $2D$, state $N=16$:

• The first is the mixer of Query, Key, Value — Dot-Product Attention from Scratch; the second is linear in $M$, $8MDN$ in all. Compare like with like: the SSM term counts only the scan, and a Mamba block also carries $D\to 2D$ projections of order $6M{D}^2$ (my count); the figure plots both.
• Table 1: Vim-Ti 76.1 against DeiT-Ti 72.2, Vim-S 80.3 against 79.8; at 1248² "Vim is 2.8× faster than DeiT and saves 86.8 % GPU memory" (Figure 1). It calls itself "the next-generation backbone for vision foundation models"; hold that thought.

Slide the width and the state size below and watch the dotted line: at 224² and patch 16 the quadratic term is a few percent of a block; nothing linear can save you much there.

model width D (heads = D/64)

768

SSM state size n

16

VMamba (Jan 2024) [LIU24]

• Swin-like stages; SS2D scans the grid along four routes and merges them.
• Table 1: VMamba-T 82.6 at 1,686 img/s, -B 83.9 at 646 img/s, +0.4 over Swin-B at 40 % more throughput; COCO 47.3 / 48.7 / 49.2 box AP, ADE20K 47.9 / 50.6 / 51.0 mIoU (Table 2).

Depth and geometry (Jan 2024 to Dec 2025) [YANG24] [DAV24] [WANG25] [LIN25] [ZHAN25]

• One lane, five papers: depth from one image, then cameras, depth and points from any number of views, on DINO patch tokens. The first mass-market consumers of frozen DINOv2 features.
• Depth Anything (Jan 2024) [YANG24]: a DINOv2 encoder with a DPT head, 1.5 M labelled plus 62 M pseudo-labelled images; zero-shot KITTI AbsRel 0.076 against MiDaS v3.1's 0.127 (Table 2). V2 (Jun 2024) [DAV24]: labels only from synthetic images, students up to 1.3 B parameters.
• VGGT (Mar 2025) [WANG25]: one feed-forward transformer that outputs cameras, depth, point maps and 3-D tracks for one to hundreds of views "in under one second" (abstract). Depth Anything 3 (Nov 2025) [LIN25]: "a single plain transformer (e.g., vanilla DINO encoder) is sufficient as a backbone"; pose accuracy +44.3 % over VGGT (abstract).
• D4RT (Dec 2025) [ZHAN25], CVPR 2026 best paper: a 1 B ViT-g video encoder builds one scene representation and a small cross-attention decoder answers queries (pixel, source frame, target frame) → 3-D point, so you decode only what you need; 9× faster than VGGT (Table 3). Transformers eating multi-view geometry.

SD3 / MM-DiT (Mar 2024) [ESSE24]

• Rectified flow, $z_t=(1-t)x_0+tϵ$; MM-DiT keeps two weight streams, text and image, with joint attention over the concatenated tokens; 0.15 B to 8 B.
• Validation loss falls smoothly with size and predicts human preference (Figure 8); GenEval 0.74 against DALL·E 3's 0.67 (Table 5). DiT's block with CLIP — Learning Pictures From the Words Next to Them-style text tokens attending alongside the image tokens.

MambaOut (May 2024) [YU24]

• Does vision need the SSM at all?
• Keep the Mamba block, delete the SSM, and call a sequence long only when $L>6D$, the crossover of An Image Is Worth 16×16 Words — ViT, Patch by Patch §4 where $2L^{2}D$ overtakes $12L{D}^2$; ImageNet at 196 tokens and $D=384$ is far below 2,304.
• MambaOut-S 84.1 against LocalVMamba-S 83.7 at 79 % of the MACs (Table 1): on classification the SSM is unnecessary. COCO 45.1 against VMamba-T 46.5 (Table 2), ADE20K 47.9 against 48.4 (Table 3): on dense tasks it still helps. The title is Kobe Bryant's farewell line.

Demystify Mamba, MILA (May 2024) [HAN24]

• Rewrite Mamba as linear attention with six differences and ablate each.
• Table 1: linear attention 77.6, +0.8 from the forget gate, +2.5 from the block design, to 80.9. MILA keeps those two, swaps the gate for a positional encoding to stay parallel, and gets 83.5 / 84.4 / 85.3 against VMamba's 82.5 / 83.6 / 83.9 at about 1.5× the speed (Table 3, Figure 6). The "what in Mamba actually matters" paper.

MambaVision (Jul 2024) [HATA24]

• CNN stages 1–2, Mamba mixers, and self-attention in the last N/2 blocks of stages 3–4.
• Table 1: -B 84.2 at 3,670 img/s, -L2 85.3; attention in the last half gives 82.3 against 81.3 for random placement (Table 5). Keep a little attention where the tokens are few.

SAM 2 (Aug 2024) [RAVI24]

• An MAE-pre-trained Hiera encoder; memory attention that cross-attends to a first-in-first-out bank of past-frame features and object pointers; the same prompt encoder and decoder plus an occlusion head; SA-V, 50.9 K videos and 35.5 M masks.
• Image 1-click mIoU 61.9 against SAM's 58.1 at 130.1 against 21.7 FPS (Table 5); SA-V J&F 77.9 for Hiera-L (Table 6); "3× fewer interactions". Video is more tokens; the memory bank is the pragmatic answer to attending over a clip.

REPA (Oct 2024) [REPA24]

• While training a DiT/SiT, regress an intermediate hidden state onto frozen DINOv2 patch features of the clean image.
• SiT-XL trains 17.5× faster; FID 1.42 with guidance on ImageNet 256² (abstract). A discriminative transformer teaching a generative one.

AIMv2 (Nov 2024) [FINI24]

• A prefix-attention encoder and a causal decoder that autoregressively predicts image patches as pixels and then caption tokens; about 12 B samples, up to 2.7 B parameters.
• ImageNet frozen-trunk attentive probe 89.5 % at 448 px against SigLIP ViT-1B's 88.1 and DINOv2 ViT-g's 87.2 (Table 3). The "predict the next patch and the next word" alternative to contrastive.

4. 2025: The Foundation Models

SigLIP 2 (Feb 2025) [TSCH25]

• SigLIP's loss plus a captioning decoder, self-distillation and masked prediction; WebLI in 109 languages; a NaFlex variant for native resolution.
• Table 1: ImageNet zero-shot 84.1 for So400m/14 at 384 px (SigLIP 83.2), 85.0 for g/16 at 384.

Web-SSL (Apr 2025) [FAN25]

• Is language supervision needed for VQA-grade features? DINOv2's recipe and MAE on the same 2 B MetaCLIP images, scaled to 7 B parameters, no text.
• Table 3: Web-DINO 7B against MetaCLIP ViT-G, ImageNet linear 86.4 against 86.4, ADE20K 52.8 against 46.7 mIoU, VQA average 59.9 against 54.8; SSL scales "nearly log-linearly" to 7 B while CLIP's VQA "largely saturates after 3B parameters" (Figure 3).

Perception Encoder (Apr 2025) [BOLY25]

• "The best visual embeddings are not at the output of the network": a contrastive encoder's best features for different tasks live in different layers.
• PE-core, contrastive on 5.4 B pairs in sizes up to 1.88 B (Table 4), then a language alignment and a spatial alignment distilled from SAM 2.1 masks that pull those intermediate features to the output.
• Table 5: ImageNet zero-shot 86.6 %, ObjectNet 88.2, ImageNet-A 92.6. The CLIP — Learning Pictures From the Words Next to Them recipe scaled, and the backbone of SAM 3.

V-JEPA 2 (Jun 2025) [ASSR25]

• Predict the latent of masked video regions from an EMA teacher; over a million hours of video; a 1.2 B ViT-g.
• Something-Something v2 77.3, Epic-Kitchens anticipation 39.7 recall@5, zero-shot pick-and-place on a Franka arm after under 62 hours of robot video (abstract). MAE's masking with DINO's latent targets, on video.

DINOv3 (Aug 2025) [SIME25]

• Long training of big SSL ViTs makes the dense features drift while global accuracy still improves.
• A ViT-7B/16, LVD-1689M from 17 B Instagram images, a million iterations, RoPE with jitter (rotary positions: the shift-is-a-rotation of Attention Is All You Need — The Transformer, Block by Block §4.3 applied to the queries and keys instead of added to the input), and Gram anchoring, a refinement phase toward an earlier teacher:

• $X_S,X_G\in {ℝ}^{P\times d}$ are the L2-normalised patch features of the student and of the earlier "Gram teacher" for the same image (the paper's Eq. 2). It pins not each patch feature but the pattern of similarities between patches, which is what dense tasks read.
• Table 3: ADE20K linear 55.9 mIoU, +6.4 over DINOv2 as DINOv3 quotes it; NYUv2 RMSE 0.309; Gram refinement alone "+2 mIoU"; then distillation into S/B/L/H+ and a LiT-style text tower. The frozen-feature state of the art at the time of writing.

NOTE: DINOv3 quotes DINOv2's ADE20K at 49.5 where DINOv2's own Table 10 says 49.0, with different heads and protocols. Cite each paper for its own numbers.

RAE (Oct 2025) [ZHEN25]

• The SD-VAE latents DiT diffuses in are low-dimensional and semantically poor.
• Freeze a representation encoder (DINOv2, SigLIP 2 or MAE) as the latent space, train a ViT decoder, and diffuse there with a wide DiT and a dimension-dependent noise shift.
• rFID 0.16 with the MAE encoder against SD-VAE's 0.62 (Table 1a); ImageNet 256² FID 1.13 with guidance (Table 8); 47× faster convergence than SiT-XL. The DiT line on ImageNet 256²: 10.94 → 3.60 → 2.27 → 1.42 → 1.13.

SAM 3 (Nov 2025) [CARI25]

• SAM segments the thing you click; it cannot find "all yellow school buses".
• Promptable concept segmentation from noun phrases and exemplars; a Perception Encoder backbone shared by a DETR-style detector and a tracker; a presence head that decides "is it here?" separately from "where"; 4 M concept labels.
• Table 1: SA-Co/Gold cgF1 54.1 against OWLv2's 24.6; LVIS zero-shot mask AP 48.5 against 38.5; 30 ms per image on an H200. The queries of DETR — Detection as Set Prediction, No Anchors and No NMS, the text of CLIP — Learning Pictures From the Words Next to Them and SAM's decoder in one model.

5. 2026 So Far

ViT-5 (Feb 2026) [WANG26]

• What does a mid-2020s plain ViT look like?
• Seven deltas: LayerScale (the γ of ConvNeXt in Convolutions Strike Back — Was It the Attention or the Recipe?), RMSNorm, 2-D RoPE plus absolute positions, four registers, QK-norm, no QKV biases, GeLU kept (SwiGLU rejected as "over-gating").
• ViT-5-B 84.2 % against 83.8 for DeiT-III-B, the 2022 update of the DeiT — Training a Vision Transformer on ImageNet Alone recipe, at 224² (Table 5); ADE20K ViT-5-L 52.0 against 49.3 (Table 8); as an SiT backbone FID 1.84 against 2.06 (Table 7). A plain ViT with everything the decade learned folded in.

V-JEPA 2.1 (Mar 2026) [MUR26]

• A dense loss on all tokens, deep supervision, a 2 B ViT-G at 64 × 384² frames.
• Table 8: NYUv2 RMSE 0.307 against DINOv3 ViT-7B's 0.309; SSv2 77.7. A video-trained encoder level with the best image-trained one on dense tasks.

SF-Mamba (Mar 2026) [YOSH26]

• MambaVision's macro-architecture, attention layers in the last stages included, with patch swapping to get bidirectional flow from one unidirectional scan and batch folding for GPU parallelism.
• Table 1: SF-Mamba-B 84.4 % at 3,534 img/s against MambaVision-B's 84.2 at 2,974 and VMamba-B's 83.9 at 640. A hybrid level with the hybrids, five times faster than the 2024 pure scans.
• The SSM line itself is alive: Mamba-3 (Mar 2026) [LAHO26] improves on Mamba-2 in language (57.6 against 55.7 at 1.5 B, Table 3) and reports no vision experiments.

Table. 1 Supervised ImageNet-1k at about 90 M parameters. ViT-5 and DeiT-III from Table 5 of [WANG26]; the SSM rows and their throughput from Table 1 of [YOSH26] (same hardware); MILA-B from Table 3 of [HAN24], measured in a different setup, so no shared throughput.

6. Where Things Stand in October 2026

Settled.

• The plain ViT won the backbone argument: DINOv3, SAM 3, Perception Encoder, SigLIP 2, V-JEPA 2.1, D4RT, VGGT and Depth Anything 3 all run on a plain or lightly modified ViT, and ViT-5 is the recipe written down.
• The pre-training target moved from pixels to latents: MAE's masking stayed; DINOv2, V-JEPA and RAE predict or diffuse in a learned representation.
• Self-supervised features are used frozen, and the best of them match text-supervised ones without seeing a word (DINOv2 86.5 against OpenCLIP 86.2; Web-DINO 86.4 against MetaCLIP 86.4, ImageNet linear).
• CLIP's softmax gave way to SigLIP's sigmoid; CLIP's idea is everywhere.
• Attention got cheap in memory through four FlashAttentions without changing a single output.
• Generation is a transformer too, DiT → MM-DiT → REPA → RAE, with the self-supervised encoders as its canvas.
• Segmentation went promptable, then concept-promptable, on DETR-style decoders.

Contested. Whether vision needs a state-space model at all.

• For: Vim's 2.8× speed and 86.8 % memory saving at 1248² (Figure 1); VMamba's +0.4 over Swin-B at 40 % more throughput; a cost linear in the tokens.
• Against: MambaOut matches or beats the vision Mambas on ImageNet with the SSM deleted (Table 1); MILA shows the gains come from a forget gate and a block design that parallel linear attention reproduces (Tables 1 and 3); the best "Mamba" backbones keep attention in their last blocks (MambaVision Table 5, 82.3 against 81.3); SF-Mamba-B at 84.4 and MambaVision-B at 84.2 sit level with a plain ViT-5-B at 84.2 and below MILA-B at 85.3; Mamba-3 reports no vision experiments; no 2025–2026 foundation model above uses an SSM backbone.
• My reading: SSMs won the long-sequence cost argument on paper and have not yet won the same-accuracy argument at vision sizes. Where tokens are few, attention plus FlashAttention is cheap enough, and the figure in Section 3 says where "few" ends, $N=6D$.

Also open: whether text is needed for VQA-grade features (Web-SSL says no at 7 B; Perception Encoder's 86.6 % says text still buys the most zero-shot), and whether the right latent for generation is a VAE's or an encoder's (RAE says the encoder's, on one paper of evidence).

That is the series. From a scalar gate on a channel in 2017 to a seven-billion-parameter ViT whose patch features you never fine-tune, the mechanism in the middle stayed the one from Query, Key, Value — Dot-Product Attention from Scratch: a softmax over dot products. Everything else was finding out what to feed it. Thanks for riding along :)

7. References

1. [DAO22] Dao, Fu, Ermon, Rudra, Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. 2022.
2. [PEEB22] Peebles, Xie. Scalable Diffusion Models with Transformers. 2022.
3. [DEHG23] Dehghani et al. Scaling Vision Transformers to 22 Billion Parameters. 2023.
4. [ZHAI23] Zhai, Mustafa, Kolesnikov, Beyer. Sigmoid Loss for Language Image Pre-Training. 2023.
5. [KIRI23] Kirillov et al. Segment Anything. 2023.
6. [OQUA23] Oquab et al. DINOv2: Learning Robust Visual Features without Supervision. 2023.
7. [RYAL23] Ryali et al. Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles. 2023.
8. [NAVIT23] Dehghani et al. Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution. 2023.
9. [DAO23] Dao. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. 2023.
10. [DARC23] Darcet, Oquab, Mairal, Bojanowski. Vision Transformers Need Registers. 2023.
11. [GU23] Gu, Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. 2023.
12. [ZHU24] Zhu et al. Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model. 2024.
13. [LIU24] Liu et al. VMamba: Visual State Space Model. 2024.
14. [YANG24] Yang et al. Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data. 2024.
15. [ESSE24] Esser et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. 2024.
16. [YU24] Yu, Wang. MambaOut: Do We Really Need Mamba for Vision? 2024.
17. [HAN24] Han et al. Demystify Mamba in Vision: A Linear Attention Perspective. 2024.
18. [DAV24] Yang et al. Depth Anything V2. 2024.
19. [HATA24] Hatamizadeh, Kautz. MambaVision: A Hybrid Mamba-Transformer Vision Backbone. 2024.
20. [SHAH24] Shah et al. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. 2024.
21. [RAVI24] Ravi et al. SAM 2: Segment Anything in Images and Videos. 2024.
22. [REPA24] Yu et al. Representation Alignment for Generation. 2024.
23. [FINI24] Fini et al. Multimodal Autoregressive Pre-training of Large Vision Encoders. 2024.
24. [TSCH25] Tschannen et al. SigLIP 2: Multilingual Vision-Language Encoders. 2025.
25. [WANG25] Wang et al. VGGT: Visual Geometry Grounded Transformer. 2025.
26. [FAN25] Fan et al. Scaling Language-Free Visual Representation Learning. 2025.
27. [BOLY25] Bolya et al. Perception Encoder. 2025.
28. [ASSR25] Assran et al. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. 2025.
29. [SIME25] Siméoni et al. DINOv3. 2025.
30. [ZHEN25] Zheng, Ma, Tong, Xie. Diffusion Transformers with Representation Autoencoders. 2025.
31. [LIN25] Lin et al. Depth Anything 3: Recovering the Visual Space from Any Views. 2025.
32. [CARI25] Carion et al. SAM 3: Segment Anything with Concepts. 2025.
33. [ZHAN25] Zhang et al. Efficiently Reconstructing Dynamic Scenes One D4RT at a Time. 2025.
34. [WANG26] Wang et al. ViT-5: Vision Transformers for The Mid-2020s. 2026.
35. [ZADO26] Zadouri et al. FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling. 2026.
36. [MUR26] Mur-Labadia et al. V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning. 2026.
37. [LAHO26] Lahoti et al. Mamba-3: Improved Sequence Modeling using State Space Principles. 2026.
38. [YOSH26] Yoshimura et al. SF-Mamba: Rethinking State Space Model for Vision. 2026.
39. [HE21] He et al. Masked Autoencoders Are Scalable Vision Learners. 2021.
40. [RADF21] Radford et al. Learning Transferable Visual Models From Natural Language Supervision. 2021.
41. [DOSO20] Dosovitskiy et al. An Image is Worth 16×16 Words. 2020.
42. [VASW17] Vaswani et al. Attention Is All You Need. 2017.
43. [LIU21] Liu et al. Swin Transformer. 2021.
44. [CARI20] Carion et al. End-to-End Object Detection with Transformers. 2020.
45. [TOUV21] Touvron et al. Training data-efficient image transformers & distillation through attention. 2021.