Topic · Training

Self-supervised learning

Everything on this site about self-supervised learning, in the order to read it: a page is listed after the pages it builds on. The figures below are the ones those pages use to explain it; each is live: move a slider and it recomputes.

In reading order

  1. Paper · read first: Convolutions Strike Back — Was It the Attention or the Recipe?, An Image Is Worth 16×16 Words — ViT, Patch by Patch
  2. Read · read first: Swin Transformer — Windows, Shifts and a Pyramid for Dense Prediction, CLIP — Learning Pictures From the Words Next to Them
  3. Paper

The figures these pages use

Pulled from the pages above. Move a slider and the figure recomputes, exactly as on its page.

From Masked Autoencoders — Hide 75 %, Encode the Rest › 2. The Pipeline, Module by Module

0.751
Hide most of the picture, encode the rest. Three ways to pick which 14 × 14 patches the encoder sees (shaded) and which it never does (white). Random sampling is the paper's default; block and grid are the alternatives of its Table 1f. Slide the mask ratio and watch the encoder's bill fall with it.

From Masked Autoencoders — Hide 75 %, Encode the Rest › 3. Why 75 % and Not 15 %?

75
Accuracy against mask ratio (Figure 5). Left: fine-tuning top-1, with the from-scratch 82.5 % as the dashed line. Right: linear probing on the frozen encoder. The linear probe peaks at 75 % and spans almost 20 points; fine-tuning is flat from 50 to 75 %.

From Vision Transformers Today — May 2022 to October 2026, Year by Year › 3. 2024: The State-Space Year

76816
Attention against a selective scan, as tokens grow. Left: multiply-adds per layer, log scale, from 64 to 65,536 tokens (224² at patch 16 is 196; 1024² is 4,096). Right: what each has to keep in memory. The dotted vertical line is N = 6D, MambaOut's definition of a long sequence: left of it the quadratic term is a minority of a block's cost.

Tags that go with it

Vision transformers 2Efficient attention 1Image classification 1Segmentation 1