Masked Autoencoders — Hide 75 %, Encode the Rest

BERT taught language models by hiding 15 % of the words in a sentence and asking the network to guess them [DEVL18]. The same trick in vision lagged for years. Masked Autoencoders (MAE) by He et al. [HE21] made it work with one recipe you can state in a single line: remove 75 % of the patches at random, run a ViT on the 25 % that is left, and ask a small decoder to paint the missing pixels back. The whole pipeline is below; the rest of this post walks it module by module.

Brush up on the pieces you already know, because I will not re-explain them:

• Patch embedding, the class token, learned positions and the ViT-B/L/H sizes: An Image Is Worth 16×16 Words — ViT, Patch by Patch. MAE's encoder is exactly a ViT.
• The transformer block itself: Attention Is All You Need — The Transformer, Block by Block.
• Why attention costs $n^2$ in the number of tokens: Query, Key, Value — Dot-Product Attention from Scratch. This post only uses that fact, in the cost arithmetic of Section 2.2.

Tighten your seatbelt; the maths is light, the numbers are the fun part :)

1. Why Did Masked Modelling Lag in Vision?

The authors name three gaps between language and images (their Introduction), and the whole design is a reply to them:

• Architecture. Convolutions "operate on regular grids and it is not straightforward to integrate 'indicators' such as mask tokens". A ViT [DOSO20] removes this: tokens are a list, and a list can have holes.
• Information density. "Languages are human-generated signals that are highly semantic and information-dense." Images are not: "a missing patch can be recovered from neighboring patches with little high-level understanding". Hide 15 % of a picture and the network learns to copy its neighbours.
• The decoder's role. In BERT the decoder "can be trivial (an MLP)" because it predicts words, which are already semantic. In vision the decoder reconstructs pixels, "hence its output is of a lower semantic level than common recognition tasks". So the decoder matters, and it should not be the encoder.

Three gaps, three answers: use a ViT, mask most of the image, and give the decoder its own small network.

2. The Pipeline, Module by Module

Symbols for the rest of the post: an image of 224 × 224 at patch 16 gives $N=196$ patches; $r$ is the mask ratio; $V$ is the set of visible patches and $M$ the masked ones; $x_i\in {ℝ}^{768}$ are the flattened pixels of patch $i$ ($16·16·3$).

2.1 Random Masking

Nothing in MAE adds noise or blur. The corruption is removal. The paper's implementation (§3, Simple implementation) is almost embarrassingly plain: shuffle the list of tokens, keep the head of the list, drop the tail.

• With the default $r=0.75$: $n_v=0.25\times 196=49$ visible patches and 147 masked ones.
• Shuffling and keeping the first $n_v$ entries is "equivalent to sampling patches without replacement". No sparse operations, no special kernels.
• Sampling is uniform at random. Removing large blocks or a regular grid are the alternatives (Table 1f), and we will see both lose.

NOTE: the paper calls MAE "a form of denoising autoencoding, but different from the classical DAE in numerous ways". The corruption is removal of whole patches, not added noise; the encoder never sees the corrupted positions; the loss covers only the removed patches; the decoder is a separate small network that is discarded.

Slide the ratio in the figure below. The right-hand readouts are the point of this whole section: the encoder's bill falls with the number of tokens it sees, and the attention products fall with its square.

mask ratio r

0.75

reshuffle (random seed)

1

2.2 The Asymmetric Encoder

The encoder is a ViT, with one change in what it is fed: "our encoder is a ViT but applied only on visible, unmasked patches ... Masked patches are removed; no mask tokens are used." Let's write what happens to a visible patch.

• $W_e$ is the linear patch embedding of An Image Is Worth 16×16 Words — ViT, Patch by Patch; $p_i$ is a positional embedding, here the fixed sine-cosine kind rather than ViT's learned one (Appendix A.1). The only other difference from a ViT is the index set: $i\in V$, not all 196.
• $E_{\theta }$ is the encoder, 24 blocks of width $D=1024$ (the D of An Image Is Worth 16×16 Words — ViT, Patch by Patch) for the ViT-L used in the ablations. Its output is a list of $n_v=49$ vectors; nothing stands in for the 147 missing patches.

Why does it matter that the encoder never meets a mask token? Two reasons, one about money and one about accuracy. Money first. One transformer block on $n$ tokens of width $D$ costs about

multiply-adds, which is Eq. 3 of An Image Is Worth 16×16 Words — ViT, Patch by Patch (projections, MLP, and the two $n^2$ products of Query, Key, Value — Dot-Product Attention from Scratch). Numbers are always satisfying to justify something, so let's put ViT-L in:

• Full image, $n=196,D=1024$: $12·196·{1024}^2=2.47\times {10}^9$ plus $2·{196}^2·1024=0.08\times {10}^9$. The quadratic term is only 3 % of a block at this resolution.
• Visible only, $n=49$: $0.62\times {10}^9$. That is 24.4 % of the full cost, i.e. 4.1× fewer FLOPs per encoder pass. At $r=0.5$ it is 2.0×, at $r=0.9$ it is 10.1×.
• The attention products alone fall by $(49/196{)}^2=1/16$, but at 196 tokens they are a small slice of the block, so the win is mostly the linear 4×, not the quadratic 16×. The quadratic saving is what you cash in at high resolution.

NOTE: the paper's measured numbers sit right where this arithmetic puts them: 3.3× FLOPs for the whole system with mask tokens in the encoder vs without (Table 1c), and 2.8× to 4.1× wall-clock (Table 2). Keep the formula; it will be used again in post 12 when the token count goes to 16,384.

Now accuracy. Two columns appear in every ablation from here on: fine-tune updates the whole encoder on ImageNet labels, linear probe freezes it and trains one linear layer on top (the probe of An Image Is Worth 16×16 Words — ViT, Patch by Patch §6.2). Table 1c puts the mask tokens into the encoder, BERT-style, and the result is not a tie:

Table. 1 Mask tokens in the encoder cost 3.3× and lose 13.9 linear-probe points (Table 1c of [HE21], ViT-L, ImageNet-1k).

The authors' reading: an encoder trained with mask tokens sees a token type at pre-training time that never appears at test time, and that gap hurts. Skipping them is cheaper and better; the paper calls it "a win-win scenario".

2.3 Mask Tokens and the Lightweight Decoder

After the encoder, the missing patches come back as placeholders. "We append a list of mask tokens to the list of encoded patches, and unshuffle this full list ... to align all tokens with their targets."

• $𝐦$ is one shared, learned vector: every masked position gets the same token.
• $p_i^{\prime }$ is a positional embedding added to all tokens at the decoder input. Without it the mask tokens "would have no information about their location in the image"; they would all be identical.
• $W_d$ maps the encoder width 1024 down to the decoder width $D^{\prime }=512$.

The decoder $D_{\varphi }$ is where the asymmetry shows: 8 blocks of width 512 against the encoder's 24 blocks of width 1024. "It only has 9 % FLOPs per token vs. ViT-L." A last linear layer turns each decoder output into pixels:

• Count it. The decoder runs on all 196 tokens but is narrow: $8\times (12·196·{512}^2+2·{196}^2·512)\approx 5.2\times {10}^9$ MACs, against $24\times 0.62\times {10}^9=14.9\times {10}^9$ for the encoder on 49 tokens. About a quarter of the forward pass.
• After pre-training the decoder is thrown away; "only the encoder is used to produce image representations for recognition".

How small can the decoder be? Table 1a and 1b answer it, and the answer depends on what you measure:

Table. 2 Decoder depth (top) and width (bottom), ViT-L, ImageNet-1k top-1 (Tables 1a and 1b of [HE21]). Bold is the default.

• A one-block decoder is enough if you are going to fine-tune: 84.8 vs 84.9. It is a poor choice for a frozen encoder: 65.5 vs 73.5.
• The paper's explanation: the last encoder layers of a deep-decoder model specialise in recognition, while with a trivial decoder they have to do the pixel work themselves. Fine-tuning fixes that; a linear probe cannot.

NOTE: linear probing and fine-tuning disagree more than once in this paper. The authors' own words: the two "are largely uncorrelated" (§4.3), and linear probing "is not well correlated with transfer learning performance, e.g., for object detection" (§5). When two columns look contradictory, that is the sentence to remember.

2.4 The Target: Pixels, Scored on the Missing Patches Only

The loss is a plain mean squared error between predicted and original pixels, and it is computed only on masked patches, "similar to BERT":

• $t_i$ is the target for patch $i$. In the plain version it is $x_i$, the raw pixels.
• Visible patches contribute nothing. Scoring every patch, the classical autoencoder way, costs about 0.5 points of accuracy (the paper's footnote 1), so the choice is "purely result-driven".

The better variant normalises each patch by its own statistics (Table 1d):

• Every patch is scored against its own contrast-normalised pixels, so a bright patch and a dark patch count the same; the network is asked for the structure inside a patch, not its average colour.
• The gain is small but free: 85.4 / 73.9 with normalisation against 84.9 / 73.5 without.

Pixels sound like a crutch next to the learned tokens of BEiT [BAO21], so the paper tried those too:

Table. 3 Reconstruction targets (Table 1d of [HE21]). Normalised pixels beat the tokenizer and need no extra network to produce them.

3. Why 75 % and Not 15 %?

Here is the number that makes MAE MAE. BERT masks 15 %; earlier masked-image work used 20–50 %; MAE uses 75 % (§4.1). The reason is the second gap of Section 1: pictures are redundant. Hide a sixth of a picture and the task is "extend the lines and textures". Hide three quarters and, in the authors' words, the model has to make "sense of the gestalt of objects and scenes, which cannot be simply completed by extending lines or textures". That "reasoning-like behavior is linked to the learning of useful representations".

Figure 5 of the paper is the whole argument in two curves. Slide across it:

mask ratio in %

75

• Linear probing peaks at exactly 75 %. Fine-tuning is flat from 50 to 75 % (84.9, 85.0, 84.9, 84.9) and 75 % is the ratio that is good for both, which is why it is the default. Both curves fall again by 90 %: higher is not always better.
• Linear probing is the sensitive one: 54.6 % at a 10 % mask ratio, 73.5 % at 75 %, a spread of almost 20 points. The frozen features of a barely-masked model are poor.
• Fine-tuning is forgiving: "a wide range of masking ratios (40–80 %) work well", and every point on the curve beats training the same ViT-L from scratch (82.5 %).

How you hide the patches matters as much as how many (Table 1f): random at 75 % gives 84.9 / 73.5; removing large blocks at 75 % collapses the linear probe to 63.9 (82.8 fine-tuned); a regular grid that keeps one patch of every four gives 84.0 / 66.0. The paper reads the two failures from the training loss: block masking at 75 % is the harder task (higher loss, blurrier reconstructions), the grid is the easier one (lower loss, sharper reconstructions), and both learn worse features than random. "Simple random sampling works the best for our MAE."

Two more cheap facts from the same ablation table:

• Augmentation (Table 1e): no augmentation at all gives 84.0 / 65.7; random crops 84.9 / 73.5; adding colour jitter hurts (84.3 / 71.9). The randomness of the mask is doing the job augmentation normally does; "our MAE behaves decently even if using no data augmentation".
• Schedule (Figure 7): 100 → 200 → 400 → 800 → 1600 epochs gives 82.3 / 83.3 / 84.3 / 84.9 / 85.1 fine-tuned and 57.3 / 64.4 / 69.7 / 73.5 / 75.1 linear. The authors "have not observed saturation of linear probing accuracy even at 1600 epochs".

Why can they afford 1600 epochs? Because the encoder sees a quarter of each image. "MAE's training time is 31 hours for 1600 epochs and MoCo v3's is 36 hours for 300 epochs" on the same ViT-L and 128 TPU-v3 cores; a two-crop contrastive method processes "200 % (two-crop) or even more" of every image, MAE processes 25 %.

Table. 4 Wall-clock pre-training time on 128 TPU-v3 cores (Table 2 of [HE21]). Measured 2.8× to 4.1×, right where the FLOP arithmetic of Section 2.2 put it.

4. Results

4.1 ImageNet-1k

Everything in this table uses ImageNet-1k images only, no labels during pre-training, then end-to-end fine-tuning (Table 3; these systems were pre-trained for 1600 epochs):

Table. 5 ImageNet-1k fine-tuning top-1 (Table 3 of [HE21]). 87.8 % was "the best accuracy among methods that use only ImageNet-1k data".

• The gain over scratch grows with the model: +1.3 for ViT-B, +3.3 for ViT-L, +3.8 for ViT-H. That is the "scalable" in the title: the bigger the ViT, the more a self-supervised signal helps it, which is what An Image Is Worth 16×16 Words — ViT, Patch by Patch found for labelled data. All of these numbers are after fine-tuning; features that are strong frozen are what DINOv2 later optimises, in Vision Transformers Today — May 2022 to October 2026, Year by Year.
• BEiT needs a separately trained dVAE tokenizer; MAE needs pixels.

4.2 Detection and Segmentation

The real test of a pre-trained backbone is a dense task. Mask R-CNN on COCO (Table 4) and UperNet on ADE20K (Table 5):

Table. 6 Transfer to COCO (Mask R-CNN) and ADE20K (UperNet) (Tables 4 and 5 of [HE21]). +4.0 box AP over supervised pre-training for ViT-L.

• Four box-AP points over supervised pre-training with the same ViT-L, and the same picture on segmentation. Pixels as a target do not make the features "low level" in any way that hurts detection.

5. What Came Next

There is a second way to learn a vision model without labels: not from the image alone, but from the text that sits next to it on the web. That is CLIP — Learning Pictures From the Words Next to Them, and it is next. MAE's own afterlife, as the encoder of Segment Anything and as the masking that DINOv2 and V-JEPA kept while swapping pixels for latents, is in the last post, Vision Transformers Today — May 2022 to October 2026, Year by Year.

6. References

1. [HE21] He, Chen, Xie, Li, Dollár, Girshick. Masked Autoencoders Are Scalable Vision Learners. 2021.
2. [DEVL18] Devlin, Chang, Lee, Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. 2018.
3. [BAO21] Bao, Dong, Piao, Wei. BEiT: BERT Pre-Training of Image Transformers. 2021.
4. [DOSO20] Dosovitskiy et al. An Image is Worth 16×16 Words. 2020.
5. [VASW17] Vaswani et al. Attention Is All You Need. 2017.