Convolutions Strike Back — Was It the Attention or the Recipe?

By the time Swin was out, the question had turned around. ViT borrowed nothing from convnets and needed 300M images; DeiT borrowed the convnet training recipe; Swin borrowed the convnet macro design, stages, downsampling, a 7×7 neighbourhood, and became a drop-in backbone. So how much of the gain was the attention, and how much was everything around it? Four papers answer from four directions, and this read lines up their tables.

The plain pipeline every one of them shares:

1. A stem that turns pixels into a token map.
2. Four stages; between stages the resolution halves and the channels double.
3. A block, repeated, made of spatial mixing (who talks to whom: attention or a convolution) followed by channel mixing (an MLP on each position), each with a residual.
4. A head.

Each paper fills the slots differently; Table 2 in §5 is that pipeline filled in four times.

Brush up ;)

• Patches and positions: An Image Is Worth 16×16 Words — ViT, Patch by Patch. Windows, shift, relative bias, patch merging: Swin Transformer — Windows, Shifts and a Pyramid for Dense Prediction. The recipe: DeiT — Training a Vision Transformer on ImageNet Alone.
• Depthwise convolutions and inverted bottlenecks: MixConv — Mixed Depthwise Convolutional Kernels from Google Brain. Squeeze-and-excitation: Attention Before Transformers — Score, Squash, Multiply. Dot-product attention: Query, Key, Value — Dot-Product Attention from Scratch.

1. PVT: A Pyramid, Still No Convolutions

Pyramid Vision Transformer [WANG21] is the earliest and purest of the four: it takes only the pyramid from convnets and keeps the block convolution-free. Its complaint about ViT [DOSO20] is that it is "columnar": one scale, a stride-16 or stride-32 output, too coarse and too costly for dense prediction.

• Four stages, each starting with its own patch embedding: P₁ = 4, then P₂ = P₃ = P₄ = 2, so strides 4 / 8 / 16 / 32 with channels 64 / 128 / 320 / 512 (Table 1 of the paper). The "progressive shrinking pyramid" follows ResNet's two rules: channels up as resolution goes down, most of the compute in stage 3 [WANG21].
• Learned position embeddings per stage; a class token only in stage 4.
• Attention stays global, but with a trick to make stage 1 affordable: spatial-reduction attention (SRA). Before attention, the keys and values are shrunk; the queries are not.

• Reading Eq. 1: $R_i$ is the reduction ratio of stage i, 8 / 4 / 2 / 1. Reshape folds each R×R patch of keys (or values) into one long vector of R²C numbers, $W^S$ projects it back to C, then a LayerNorm. It is a strided, non-overlapping patchify applied to K and V only; Q keeps full resolution, so the output does too.
• The attention term's cost falls by $R_i^2$: 64× at stage 1. But that is a division, not a change of exponent; Swin's paper notes PVT's complexity "is still quadratic to image size" [LIU21].

The numbers (Table 2 of the paper, ImageNet-1k, DeiT recipe; the paper reports top-1 error, converted here): PVT-Tiny 13.2M / 1.9G / 75.1, PVT-Small 24.5M / 3.8G / 79.8, PVT-Medium 44.2M / 6.7G / 81.2, PVT-Large 61.4M / 9.8G / 81.7 [WANG21]. The pyramid pays where it was built to: with RetinaNet (Table 3), PVT-Small gets 40.4 AP against ResNet-50's 36.3 (+4.1), and PVT-Tiny 36.7 against ResNet-18's 31.8; with DETR on top (Table 6), 34.7 AP from a detector with no convolution anywhere [WANG21].

2. CvT: Convolutions Inside the Embedding and the Projection

CvT [WU21] wants both lists at once: the convnet's "shift, scale, and distortion invariance" and the transformer's "dynamic attention, global context, and better generalization". It puts convolutions in two places and leaves the attention itself global.

Convolutional token embedding. Instead of chopping the image into non-overlapping patches, slide an overlapping s×s convolution with stride s − o and padding p over the previous stage's token map (laid out as 2-D), then LayerNorm. The token count falls and the width grows, like a convnet stem:

• Reading Eq. 2: o > 0 is what makes the patches overlap. Stage 1 uses 7×7 with stride 4 (a 224 px image becomes 56×56 tokens), later stages 3×3 with stride 2 [WU21].

Convolutional projection. Q, K and V come not from a linear layer but from a depthwise-separable convolution on the 2-D token map:

• Reading Eq. 3: a depthwise 3×3 followed by a 1×1. The extra cost per projection is s²C parameters, against s²C² for a dense convolution of the same size [WU21]. K and V use stride 2 in the depthwise layer, so there are 4× fewer keys and values; Q keeps stride 1.
• With convolutions in the block, CvT uses no position embedding at all, and a class token only in the last stage.

Numbers (Tables 2 and 3 of the paper, ImageNet-1k, 224 px): CvT-13, three stages of 1 / 2 / 10 blocks at widths 64 / 192 / 384, is 20M / 4.5G / 81.6; CvT-21 is 32M / 7.1G / 82.5; DeiT-S and PVT-Small sit at 79.8, DeiT-B at 81.8 [WU21]. The ablations are the interesting part:

• Position embedding removed (Table 5): DeiT-S falls 79.8 → 78.0 (−1.8); CvT-13 goes 81.5 → 81.6. Once convolutions are in the block, the position is already in the features.
• Convolutional token embedding vs non-overlapping patches, both without position embeddings (Table 6): 80.7 → 81.6.
• Convolutional projection in 0 / 1 / 2 / 3 stages (Table 8): 80.6 / 80.8 / 81.0 / 81.6.
• K/V stride 1 → 2 (Table 7): 82.3 → 81.6 for 6.55 → 4.53 GFLOPs. Shrinking the keys is an efficiency trick (−0.7 for −31 % FLOPs), not where the accuracy comes from.

3. CoAtNet: One Attention That Contains a Convolution

CoAtNet [DAI21] ("pronounced 'coat' nets") states the trade-off in a table of its own: translation equivariance belongs to convolution; input-adaptive weighting and a global receptive field belong to attention. Transformers have more capacity, but "their generalization can be worse than convolutional networks due to the lack of the right inductive bias" [DAI21]. Two ideas follow.

3.1 Relative Attention

Let's build it step-by-step, as the paper does. A depthwise convolution over a local window $ℒ(i)$ around position i, with one D-vector of weights per offset:

Self-attention over the whole map $𝒢$ is Eq. 4 of Query, Key, Value — Dot-Product Attention from Scratch with $Q=K=V=x$: the weights are a softmax over $x_i^{\top }{x}_j$.

Sum the two scores inside the softmax, a scalar $w_{i-j}$ per offset, over the whole map:

• Reading Eq. 5 against Eq. 4: the attention weight is now "decided jointly by the $w_{i-j}$ of translation equivariance and the input-adaptive $x_i^{\top }{x}_j$" [DAI21]. Convolution is the seating chart alone; attention is the conversation alone; relative attention is both.
• Making w a scalar instead of a D-vector is what lets the "kernel" be global without exploding the parameter count. The term is input-independent, not shared across layers, and not windowed. It is the same family as Swin's bias table B, but Swin's lives inside a 7×7 window and CoAtNet's spans the map. Multi-head Q/K/V projections are used but left out of the notation.
• Table 6 of the paper prices it: removing relative attention takes CoAtNet-2 from 84.1 to 83.8 on ImageNet-1k, and from 87.9 to 87.4 after ImageNet-21k pre-training, "not in higher capacity but in better generalization" [DAI21].

3.2 The Vertical Layout

Global attention is too expensive at full resolution, so the paper downsamples first (local windows were rejected as slow on TPU). Five stages, S0 to S4: S0 is a two-layer convolutional stem, S1 is MBConv with squeeze-and-excitation, and each of S2 to S4 is either a convolution stage (C) or a transformer stage (T), with the rule that convolution stages come first. MBConv is chosen because its inverted bottleneck, expand 4×, mix, project back, has the same shape as the transformer's FFN [DAI21].

• Candidates: C-C-C-C, C-C-C-T, C-C-T-T, C-T-T-T, and ViT_REL (a ViT stem with relative attention).
• Generalisation, measured as the train–test gap on ImageNet-1k without regularisation (Fig. 1): C-C-C-C ≈ C-C-C-T ≥ C-C-T-T > C-T-T-T ≫ ViT_REL. More convolution, better generalisation.
• Capacity, measured as the fit on JFT (Fig. 1): C-C-T-T ≈ C-T-T-T > ViT_REL > C-C-C-T > C-C-C-C. More attention, more capacity, until it saturates.
• The two rankings cross at C-C-T-T, which also transfers best after JFT (Table 2: 84.23 against C-T-T-T's 84.02 at 384 px). Moving four blocks from the convolution stage S2 to the attention stage S3, [2, 2, 6, 14, 2] → [2, 2, 2, 18, 2], costs 84.1 → 83.4 (Table 7) [DAI21].

Numbers (Table 4 of the paper, ImageNet-1k only, 224 px): CoAtNet-0 25M / 4.2G / 81.6, CoAtNet-1 42M / 8.4G / 83.3, CoAtNet-2 75M / 15.7G / 84.1, CoAtNet-3 168M / 34.7G / 84.5; CoAtNet-3 at 512 px reaches 86.0 with ImageNet-1k alone. With 21k pre-training, CoAtNet-3 gets 87.6 at 224 px and CoAtNet-4 reaches 88.56 at 512 px, matching ViT-Huge pre-trained on JFT-300M with "23× less data" [DAI21]. Swapping BatchNorm for LayerNorm in the convolution stages changes nothing: 84.1 → 84.1 (Table 8).

4. ConvNeXt: Take Swin's Design, Remove the Attention, Keep the Score

ConvNeXt [LIU22] runs the cleanest experiment of the four. Its abstract puts the suspicion plainly: "the effectiveness of such hybrid approaches is still largely credited to the intrinsic superiority of Transformers, rather than the inherent inductive biases of convolutions." So: start from ResNet-50, "gradually 'modernize'" it toward Swin-T at a constant ~4.5 GFLOPs, "without introducing any attention-based modules", and write the score down after every step. A ship-of-Theseus experiment with a scoreboard.

Table. 1 The ConvNeXt roadmap, ImageNet-1k top-1 averaged over three seeds (Table 10 and Fig. 2 of [LIU22]). Every step after the first names the Swin or DeiT design it copies.

Drag the step slider through the roadmap and watch the gap to Swin-T close, then go negative; the kernel slider changes only step 8, the way the paper swept it.

steps applied

13

kernel at step 8

7

What the roadmap says, step by step:

• The recipe alone is +2.7, with nothing in the network changed. The twelve architecture steps together are +3.2. Half of the "transformer gap" at this size was never about the architecture.
• The two biggest architecture steps are width 64 → 96 (+2.2, paid for in FLOPs until the inverted bottleneck claws them back) and one activation per block (+0.65). The famous 7×7 kernel is +0.65 over 3×3 after the convolution was moved up, and 9 and 11 add nothing. The lesson is "7 ≈ Swin's window", not "bigger kernels".
• BN → LN is +0.06, and CoAtNet found BN = LN. Nobody should credit LayerNorm.
• The ResNet-200 → ConvNeXt-B track (Table 11, ~15 GFLOPs) runs the same way: 78.20 → 81.14 from the recipe (+2.9) → 83.60 at the end, against Swin-B's 83.50; there, kernels 5 / 7 / 9 / 11 give 82.32 / 82.30 / 82.27 / 82.18, so 7 is kept for consistency, not for a gain [LIU22].

The block that comes out the other end (Fig. 4 of the paper), written as one line:

• Reading Eq. 6: spatial mixing first (a depthwise 7×7, standing where MSA stands), then channel mixing (two 1×1s with one GELU, the MLP), one norm, one residual. γ is Layer Scale, a per-channel scalar initialised at 10⁻⁶. Why a depthwise conv can stand in for attention, in the paper's words: it "is similar to the weighted sum operation in self-attention, which operates on a per-channel basis, i.e., only mixing information in the spatial dimension" [LIU22]. No attention, no position encoding, no mask.
• ConvNeXt-T has C = (96, 192, 384, 768) and blocks (3, 3, 9, 3); -S has 27 blocks in stage 3; -B, -L and -XL start C at 128, 192 and 256. Training: AdamW at 4·10⁻³, batch 4096, 300 epochs, weight decay 0.05, EMA [LIU22].

The results (Table 1 of the paper, ImageNet-1k only, 224 px unless stated): ConvNeXt-T 29M / 4.5G / 82.1 against Swin-T 81.3; -S 83.1 against Swin-S 83.0; -B 83.8 against Swin-B 83.5; -B↑384 85.1 against 84.5, at 95.7 vs 85.1 im/s. With 22k pre-training: -B↑384 86.8 vs Swin-B 86.4, -L↑384 87.5 vs Swin-L 87.3, -XL↑384 87.8. Dense prediction holds too: Cascade Mask R-CNN ConvNeXt-T 50.4 box / 43.7 mask, identical to Swin-T, and -B 52.7 / 45.6 vs 51.9 / 45.0 (Table 3); ADE20K -T 46.7 vs 45.8 mIoU (Table 4) [LIU22].

NOTE: the block alone matches ViT without any pyramid. Table 2 of the paper builds isotropic ConvNeXts in ViT's layout, no stages, no downsampling: ConvNeXt-S / B / L (iso.) score 79.7 / 82.0 / 82.6 against ViT-S / B / L's 79.8 / 81.8 / 82.6 at matched sizes.

5. The Four Blocks Side by Side

Table. 2 The pipeline of Fig. 1 filled in four times, with Swin for reference.

6. The Scoreboard

ImageNet-1k only, 224 px, 300 epochs, no distillation, each paper's own table:

Table. 3 Top-1 by compute tier: Table 2 of [WANG21], Table 3 of [WU21], Table 4 of [DAI21], Table 1 of [LIU22], Table 1 of [LIU21], Table 5 of [TOUV21]. PVT-Large is the family's biggest model and sits below its tier; CvT has no ImageNet-1k-only model near 15G.

Switch the tier in the figure, and switch the bars to FLOPs or parameters to see what each point of accuracy was paid with:

compute tier (1 ≈ 4.5 GFLOPs, 2 ≈ 8 G, 3 ≈ 15 G)

1

bars show (0 = top-1 above ResNet-50, 1 = GFLOPs, 2 = params M)

0

NOTE: cross-paper numbers are not one experiment. Same dataset and epoch count, but PVT trains with batch 128, CvT with batch 2048 and learning rate 0.02, CoAtNet with RandAugment and Mixup as its only augmentations, ConvNeXt with batch 4096, EMA and Layer Scale, and each counts FLOPs its own way. Read tiers, not decimals.

7. So, Is It the Attention or the Recipe?

Adding up what the four tables agree on:

• Recipe first. +2.7 on ResNet-50 before touching the network (ConvNeXt Table 10).
• Pyramid second. PVT and Swin both beat the columnar ViT on dense tasks by building strides 4 to 32; PVT shows a pyramid needs no convolutions, ConvNeXt shows it needs no attention.
• The block shape matters more than what fills it. Inverted bottleneck, one norm, one activation, spatial mixing before channel mixing: ConvNeXt's +3.2 of architecture is mostly these, and CoAtNet picked MBConv for the same reason. The isotropic ConvNeXt matching ViT is the sharpest version of the point.
• Convolutions buy generalisation, attention buys capacity. CoAtNet's two rankings cross at C-C-T-T; putting attention early costs 0.7. Once convolutions are in the block, position embeddings become optional (CvT), and a translation prior inside attention is worth +0.3 to +0.5 globally and +1.2 in windows.
• At the ~4.5 G tier the ranking is ConvNeXt-T 82.1 > CvT-13 = CoAtNet-0 81.6 > Swin-T 81.3 > PVT-Small = DeiT-S 79.8. The network with zero attention is on top; the ones with the most attention are at the bottom. At ~15 G, CoAtNet-2's global attention after three convolution stages edges ahead, 84.1 to 83.8.

My reading is that by early 2022 the architecture argument had reached a draw. With the same recipe, the same pyramid and the same block shape, a 7×7 depthwise convolution and a 7×7 attention window land within a point of each other, in either direction, depending on size.

8. What Came Next

If the architecture lever is spent, the remaining lever is what the network is trained on. The next post, Masked Autoencoders — Hide 75 %, Encode the Rest, takes the plain ViT that this whole read has been improving around and changes the pre-training instead: hide three quarters of the patches and ask the network to paint them back. Stay tuned.

9. References

1. [WANG21] Wang, Xie, Li, Fan, Song, Liang, Lu, Luo, Shao. Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions. arXiv:2102.12122.
2. [WU21] Wu, Xiao, Codella, Liu, Dai, Yuan, Zhang. CvT: Introducing Convolutions to Vision Transformers. arXiv:2103.15808.
3. [DAI21] Dai, Liu, Le, Tan. CoAtNet: Marrying Convolution and Attention for All Data Sizes. arXiv:2106.04803.
4. [LIU22] Liu, Mao, Wu, Feichtenhofer, Darrell, Xie. A ConvNet for the 2020s. arXiv:2201.03545.
5. [LIU21] Liu, Lin, Cao, Hu, Wei, Zhang, Lin, Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. arXiv:2103.14030.
6. [TOUV21] Touvron, Cord, Douze, Massa, Sablayrolles, Jégou. Training data-efficient image transformers & distillation through attention. arXiv:2012.12877.
7. [DOSO20] Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010.11929.