Attention Before Transformers — Score, Squash, Multiply

Everybody says attention and means something slightly different. Before the transformer there were already three things called attention, and they look nothing alike on paper: one lives between an RNN encoder and decoder, one is a tiny MLP bolted on a ResNet block, one is an $N\times N$ matrix the size of a feature map squared. The problem statement of this read is to make them look alike, because they are.

Every attention mechanism before the transformer does the same three things: score how relevant each piece of the input is, squash the scores into weights, and multiply the weights back into the features. What changes between papers is what gets scored (a source word, a channel, a pair of pixels) and what the score is a function of.

This is the first read of the Transformers in Vision series, so there is little to brush up on. Skim my write-up of Understanding Attention Modules: CBAM and BAM — A Quick Read if you have not: BAM and CBAM are the channel gate of Section 3 with a spatial gate next to it, and I will not re-derive them here. Residual blocks I assume you know. The maths is light but the numbers are many :)

1. The Template: Score, Squash, Multiply

Let me state the recipe once, so that every later section is just filling in blanks.

• Score. For each piece of the input (a word, a channel, a position), one number saying how relevant it is relative to something: a decoder state, the other channels, another position.
• Squash. Turn the scores into weights: a softmax across the pieces (weights compete and sum to 1) or an independent sigmoid per piece (gates in $(0,1)$ that do not compete).
• Multiply. Scale the features by their weights and, if the pieces are many, sum them up.

Watch the last column of Fig. 1. When the pieces are pairs of positions and the score depends on the content at both, the recipe stops being a gate and becomes a conversation.

2. Bahdanau 2014 — Attention as a Soft Search

2.1 The Bottleneck

The RNN encoder-decoder of 2014 read a sentence into one vector and wrote the translation out of it. The encoder must "compress all the necessary information of a source sentence into a fixed-length vector", and the authors "conjecture that the use of a fixed-length vector is a bottleneck" [BAHD14]. Their Fig. 2 is the evidence: BLEU drops steeply as sentences get longer.

The fix is to stop compressing. Keep every encoder state and let the decoder pick, at every output word, which source words to look at. The paper calls this a (soft-)search: the decoder "(soft-)searches for a set of positions in a source sentence where the most relevant information is concentrated" [BAHD14]. Think of a translator who keeps a finger on the source text and slides it along as they speak, instead of reading the page once and reciting from memory.

2.2 Let's Devise It Step by Step

Start from the plain encoder-decoder: the decoder state $s_i$ is updated from the previous state, the previous word and one fixed context vector $c$, the same $c$ for the whole sentence:

The first change is tiny on paper and is the whole idea: let the context depend on $i$.

Now $c_i$ must be built from the encoder. A bidirectional RNN gives one annotation $h_j\in {ℝ}^{2n}$ per source word (n = 1000 units each way, §4.2 of [BAHD14]), and $c_i$ is a weighted average of all of them. The three lines below are the template of Section 1 written out (Eqs. 5–6 of the paper and the alignment model under them):

• Score. $e_{ij}$ says how well source word $j$ matches the decoder's current state $s_{i-1}\in {ℝ}^n$. The paper calls $a(·)$ the alignment model: a one-hidden-layer MLP with $W_a\in {ℝ}^{n\times n}$, $U_a\in {ℝ}^{n\times 2n}$ and $v_a\in {ℝ}^n$ (App. A.1.2). Every (target step, source word) pair gets a score: $T_x\times T_y$ of them per sentence pair.
• Squash. The softmax turns a row of scores into weights ${\alpha }_{ij}\in (0,1)$ that sum to 1 over the source words. Nothing is pooled or thrown away.
• Multiply. $c_i\in {ℝ}^{2n}$ is the average of the annotations under those weights, "the expected annotation" in the paper's words. Attention "relieve[s] the encoder from the burden of having to encode all information… into a fixed-length vector" [BAHD14].

2.3 The Numbers

Numbers are always satisfying to justify something. Table 1 of the paper is WMT'14 English→French test BLEU; models are named by the longest training sentence.

Table. 1 WMT'14 En→Fr test BLEU, from Table 1 of [BAHD14].

• RNNsearch-30, which never saw a sentence longer than 30 words in training, beats RNNencdec-50. That is the bottleneck argument in one row.
• Their Fig. 2: RNNsearch-50 "shows no performance deterioration even with sentences of length 50 or more".
• Their Fig. 3 draws the ${\alpha }_{ij}$ matrices in grey-scale, source words on one axis, target words on the other: the first attention map, and every heat map in this series descends from it.

NOTE: The paper's word for these matrices is alignment. They are the weights ${\alpha }_{ij}$, nothing more; reading them as "what the model looks at" in a causal sense is our habit, not theirs.

2.4 Where It Sits, and One Shortcut

Between the encoder and the decoder: the thing being compared is a decoder state, the things being averaged are encoder annotations.

NOTE: This is cross-attention, not self-attention. Nothing in the source sentence is scored against anything else in the source sentence yet.

A year later Luong, Pham and Manning asked whether the MLP in the score was needed and tried plain products (§3.1 of [LUON15]):

Same softmax, same weighted sum; only the first line of Eq. 3 changed ($h_t$ is the target state, ${\overline{h}}_s$ a source state). File this away: the dot product is the bridge to the next read.

3. Squeeze-and-Excitation — Attention over Channels

3.1 Why Gate Channels?

A convolution mixes space and channels inside one local receptive field; nothing in a plain ResNet lets a layer use global context to decide which channels matter for this image. Squeeze-and-Excitation [HU17] calls the fix feature recalibration: "use global information to selectively emphasise informative features and suppress less useful ones" (§1), a gating mechanism (§3.2). The picture in my head is a mixing desk: one fader per channel, set after listening to the whole mix.

3.2 Let's Devise It Step by Step

Take the output of a residual branch, $U\in {ℝ}^{H\times W\times C}$, one map $u_c$ per channel. Eqs. 2–4 of the paper are the three steps:

• Squeeze. Global-average-pool each channel to a single number; $z\in {ℝ}^C$ is a summary of the whole image, one entry per channel.
• Excitation. A two-layer MLP looks at all $C$ numbers at once and outputs one gate per channel: $W_1\in {ℝ}^{(C/r)\times C}$ squeezes to width $C/r$, $\delta$ is a ReLU, $W_2\in {ℝ}^{C\times (C/r)}$ expands back, $\sigma$ is a sigmoid, so $s\in (0,1{)}^C$. The reduction ratio is r = 16.
• Scale. Multiply every map by its gate. The scaled $\tilde{x}$ goes on to the residual add.

Compare with Eq. 3 and two things have changed. The score for channel $c$ depends on the whole input through $z$, not on a pair. And the squash is a sigmoid, so the $C$ gates are independent: they neither compete nor sum to 1. All gates at 1 is the identity.

Let's watch the gate work on a toy map of eight channels. Slide β and see the gates separate:

gate sharpness β

2

log₂ C of the real block

8

At β = 0 every gate is 0.5 and the block halves the whole map; as β grows the gates separate into a hard on/off per channel. The real block learns where on that axis to sit, per channel, per image.

3.3 The Numbers

How much does the gate cost? Eq. 5 of the paper counts the two FC layers over S stages, with $N_s$ blocks and $C_s$ channels in stage s (biases ignored):

• One block at C = 256, r = 16. $W_1$ is 256 × 16 = 4,096 weights, $W_2$ is 16 × 256 = 4,096: 8,192 weights ($2·{256}^2/16$) plus 272 biases. A 3×3 convolution at that width has 256 × 256 × 9 = 589,824 weights, so the gate costs 1.4 % of one conv.
• SE-ResNet-50. Summing Eq. 6 over the four stages gives ≈ 2.5 M on top of ResNet-50's ~25 M, the ~10 % of §4 [HU17]; 63 % of it sits in the last stage, which is why the last-stage blocks can be dropped cheaply.
• FLOPs barely move: 3.86 → 3.87 GFLOPs (+0.26 %).

Table. 2 ImageNet validation, single crop, from Table 2 of [HU17].

• SE-ResNet-50 (6.62 top-5) "approach[es]" ResNet-101 (6.52) at half the FLOPs; SE-ResNet-101 (6.07) beats ResNet-152 (6.34) (§5.1 of [HU17]).
• The reduction ratio hardly matters: Table 10 has r = 2 … 32 all between 22.25 and 22.72 top-1 error against 23.30 plain; r = 16 (28.1 M params) is the chosen balance.
• Dropping the squeeze (Table 16) costs 22.93 vs 22.28: the global pooling does work, not just the MLP.
• The SENet entry won ILSVRC 2017 with 2.251 % top-5 error.

3.4 Where It Sits

After the residual branch and before the add, once per block (Fig. 2). Along the depth (§7.2, their Fig. 7) the early gates are nearly the same for every class, the gates at SE_4_6 and SE_5_1 become class-specific, SE_5_2 saturates with most gates near 1, the identity, and SE_5_3 shows one pattern for every class up to a scale. The paper reads this as the last stage mattering least, hence the cheap removal above.

NOTE: SE is not self-attention. The gate depends on pooled statistics, not on pairs; it is a sigmoid, not a softmax; the C gates never compete. The paper's own words are "a simple self-gating mechanism" (§1) and "a lightweight gating mechanism" (§2) [HU17].

SE's channel gate plus a spatial gate is BAM and CBAM [PARK18] [WOO18]; everything about them is in Understanding Attention Modules: CBAM and BAM — A Quick Read.

4. Non-local Neural Networks — Pairwise Attention over Positions

4.1 Why Compare Every Position with Every Other?

"Convolutional and recurrent operations both process a local neighborhood… long-range dependencies can only be captured when these operations are applied repeatedly" [WANG17]: "computationally inefficient", prone to "optimization difficulties", and "messages need to be delivered back and forth between distant positions" (§1). The fix is a block in which every position reads every other position in one step, "a generalization of the classical non-local means" filter that averages similar patches from anywhere in the image. If SE is a mixing desk, the non-local block is a group chat: every pixel reads every other pixel's message and weighs it by how much it sounds like its own.

4.2 Let's Devise It Step by Step

Flatten the feature map to $N$ positions ($N=THW$ for video, $HW$ for an image), each with a feature $x_i\in {ℝ}^C$. Eqs. 1, 3 and 6 of the paper, in the embedded-Gaussian form:

• Score. Compare position $i$ with every position $j$: embed both with 1×1 convolutions ($W_{\theta },W_{\varphi }\in {ℝ}^{(C/2)\times C}$), take the dot product, exponentiate. Every ordered pair $(i,j)$ gets a score that depends on the content at both ends.
• Squash. Divide by the row sum $C(x)$. The authors note that "$(1/C(x))f$ becomes the softmax along j", so the whole thing is $y=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(x^{\top }{W}_{\theta }^{\top }{W}_{\varphi }x)g(x)$, "the self-attention form in [49]" [WANG17], and reference 49 is the transformer paper. That sentence makes this block the hinge of the series.
• Multiply. Average the projected features $g(x_j)\in {ℝ}^{C/2}$ under those weights, project back up with $W_z\in {ℝ}^{C\times (C/2)}$, add the input.

In matrix form (their Fig. 2): $Y=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(\theta {\varphi }^{\top })G$ with $\theta ,\varphi ,G\in {ℝ}^{N\times C/2}$; the thing in the middle is $N\times N$.

Here is one row of that $N\times N$ matrix on a toy image. Slide the query along the middle row and watch who it listens to:

query column (row 12)

5

temperature τ

0.15

The temperature slider (the next read names this knob) stands in for the scale of $\theta$ and $\varphi$: small τ picks one or two positions, large τ averages the whole image. And the plain dot product rewards bright pixels rather than similar ones, which is what the learned $W_{\theta },W_{\varphi }$ are there to fix.

4.3 Does the Softmax Matter?

The paper tries four pairwise functions $f$ (Table 2a, Kinetics top-1, one block on a ResNet-50 C2D): Gaussian $\exp (x_i^{\top }{x}_j)$ 72.5, embedded Gaussian 72.7, dot product with $C(x)=N$ and no softmax at all 72.9, concatenation 72.8. Their conclusion: "the attentional behavior (due to softmax) is not essential" [WANG17].

NOTE: Only the embedded-Gaussian version is softmax attention. The dot-product version divides by N, has no softmax, and works just as well. "Non-local = self-attention" is true of one version out of four.

4.4 The Numbers

• Parameters, one block on a 1024-channel map. $W_{\theta },W_{\varphi },W_g$: 3 × 1024 × 512 = 1,572,864; $W_z$: 512 × 1024 = 524,288; 2,097,152 ≈ 2.1 M.
• The pairwise cost at the paper's size, T = 4 frames of 14 × 14: N = 784, so the affinity matrix has 784² = 614,656 entries (2.46 MB in fp32). $\theta {\varphi }^{\top }$ costs 614,656 × 512 ≈ 315 M multiply-adds, multiplying by G another 315 M, the four 1×1 projections ≈ 1.64 G. Here the projections dominate, not the pairs; a 2×2 max-pool after φ and g (§3.3) cuts the pairs by 4× more.
• Why nobody did this on pixels. At 224 × 224, N = 50,176 and $N^2=2.52\times {10}^9$ affinities: 10 GB in fp32 for one map. Attention Meets Pixels — Windows, Axes and the First Fully Attentional Networks starts from this number, so I say it once and stop.

Table. 3 Kinetics top-1 accuracy, from Table 2 of [WANG17].

• One block on a ResNet-50 gives +0.9 on Kinetics; ten give +2.5.
• On COCO (Table 5 of the paper, Mask R-CNN, one block) AP^box goes 38.0 → 39.0 on ResNet-50 and 44.1 → 45.0 on ResNeXt-152, AP^mask 34.6 → 35.5 on ResNet-50, all for "<5 % computation" [WANG17].
• Their Fig. 1 and Fig. 3 show the weights on video: the ball in the first frame is related to "the ball in the last two frames". Content found content, across time, in one step.

4.5 Where It Sits

• Inside a residual unit, with the BN after $W_z$ zero-initialised, so the block starts as the identity ($z_i=x_i$) and drops into a pre-trained network without disturbing it (§4.1). SE and non-local are plug-and-play for the same reason: they sit on a skip and begin as "do nothing".
• Not everywhere. Table 2b puts one block in res2 / res3 / res4 / res5 and gets 72.7 / 72.9 / 72.7 / 72.3; res5 gains least because a 7 × 7 map leaves little to look at. The paper uses 1, 5 or 10 blocks in res3 and res4.

5. Three Mechanisms, One Recipe

Closing the loop on Fig. 1: in Bahdanau's attention the thing being compared (a decoder state) and the things being averaged (encoder annotations) come from two different sequences. In SE nothing is compared with anything; a summary of the whole input sets a gate per channel. In the non-local block both come from the same feature map, every position against every position, softmax across them. That is self-attention, running inside a ResNet months after the transformer paper, with the authors saying so. The next read gives the two things their proper names. Two? Actually three, thanks to a small trick with projections, but let's wait for that ;)

6. What Came Next

Luong showed that the score in Eq. 3 can be a plain dot product and nothing is lost. Vaswani and colleagues then asked what happens if the dot product is all you keep: no RNN, no convolution, just scores, a softmax and a weighted sum, run many times in parallel. That is Query, Key, Value — Dot-Product Attention from Scratch, and it starts where Eq. 4 and Eq. 7 leave off.

7. References

1. [BAHD14] Bahdanau, Cho, Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015.
2. [LUON15] Luong, Pham, Manning. Effective Approaches to Attention-based Neural Machine Translation. EMNLP 2015.
3. [HU17] Hu, Shen, Albanie, Sun, Wu. Squeeze-and-Excitation Networks. CVPR 2018.
4. [WANG17] Wang, Girshick, Gupta, He. Non-local Neural Networks. CVPR 2018.
5. [PARK18] Park, Woo, Lee, Kweon. BAM: Bottleneck Attention Module. BMVC 2018.
6. [WOO18] Woo, Park, Lee, Kweon. CBAM: Convolutional Block Attention Module. ECCV 2018.