Topic · Mechanisms

Attention modules

Everything on this site about attention modules, in the order to read it: a page is listed after the pages it builds on. The figures below are the ones those pages use to explain it; each is live: move a slider and it recomputes.

In reading order

  1. Paper · read first: YOLOv4 — Version 1: Bag of Freebies
  2. Paper
  3. Read
  4. Paper
  5. Paper

The figures these pages use

Pulled from the pages above. Move a slider and the figure recomputes, exactly as on its page.

From YOLOv4 — Version 2: Bag of Specials › 1. Mish Activation

-1.0
Graph of the ReLU function. ReLU is a ramp that starts at 0. Left of 0 both the output and the slope are zero.

From YOLOv4 — Version 2: Bag of Specials › 1. Mish Activation

-1.2
Graph of the Mish function. Mish dips a little below zero for negative inputs instead of cutting them off, then follows the input.

From YOLOv4 — Version 2: Bag of Specials › 1. Mish Activation

-1.0
Mish against ReLU and Swish. Mish is smooth everywhere and lets small negative values through instead of cutting them to zero.

From YOLOv4 — Version 2: Bag of Specials › 2. CSP(Cross Stage Partial Connections)

166
What each dense layer reads. Input channels seen by each layer of a dense block. The same early channels (and their gradients) are reused by every later layer, which is the duplication CSP removes.

From YOLOv4 — Version 2: Bag of Specials › 6. DIoU NMS⁹

140.5
Keep or suppress, by DIoU. The dashed box M is the one already kept. Slide the candidate B_i across it: ordinary NMS looks only at IoU, DIoU-NMS also asks how far apart the centres are.

From Understanding Attention Modules: CBAM and BAM — A Quick Read › 2. Bottleneck Attention Modules(BAM)

1.0
F' = F + F ⊗ M(F) on a feature map. An 8×8 feature map with two blobs, an attention map that favours the right one, and the result. Attention can only amplify.

From Understanding Attention Modules: CBAM and BAM — A Quick Read › 2. Bottleneck Attention Modules(BAM)

1.0-2.0
σ(M_c + M_s). The two branch scores add before the sigmoid, so either branch can push a position open on its own, and both together saturate.

From Understanding Attention Modules: CBAM and BAM — A Quick Read › 2. Bottleneck Attention Modules(BAM)

512
Cost of channel attention. A larger reduction ratio makes the attention module cheaper, at the price of a narrower bottleneck.

From Understanding Attention Modules: CBAM and BAM — A Quick Read › 2. Bottleneck Attention Modules(BAM)

2
What the spatial branch can see. Side of the window one output pixel sees, for each dilation value. Dilation widens the view without adding weights.

From Understanding Attention Modules: CBAM and BAM — A Quick Read › 3. Convolutional Block Attention Module(CBAM)

0.80.8
Channel attention, then spatial attention. Rows are channels, columns are positions. M_c rescales whole rows, M_s rescales whole columns. The product of the two gates is what survives.

From Understanding Attention Modules: CBAM and BAM — A Quick Read › 3. Convolutional Block Attention Module(CBAM)

0.9
Average pool and max pool see different channels. Eight channels of one feature map. Channel 6 is quiet everywhere except one spike: the average barely notices, the max does.

From Understanding Attention Modules: CBAM and BAM — A Quick Read › 3. Convolutional Block Attention Module(CBAM)

7
Where to look. Channel-wise average and max maps of a 12×12 feature, and the gate a k×k filter produces from them.

From Attention Before Transformers — Score, Squash, Multiply › 3. Squeeze-and-Excitation — Attention over Channels

28
A gate per channel, applied to the whole channel. Rows are the 8 channels, columns the 16 positions of a 4×4 map. The middle column is the gate vector s; the real block gets it from a learned 2-layer MLP, here it is a sigmoid of how far each channel's mean sits from the others. β = 0 gives every channel 0.5; large β drives the gates to 0 or 1.

From Attention Before Transformers — Score, Squash, Multiply › 4. Non-local Neural Networks — Pairwise Attention over Positions

50.15
One pixel against every other pixel. A 24×24 toy image: a bright blob on the left (col 5), a mid-grey blob in the middle (col 12) and a copy of the bright blob on the right (col 19). The marked cell is the query; every cell is shaded by the weight the query gives it. Slide the query through the left blob and watch its far-away twin light up. Put it on the grey blob and the bright blobs win instead: a plain dot product rewards brightness, not sameness, which is why the paper puts learned θ and φ in front of it.

Tags that go with it

Activation functions 1Backbone design 1Convolutions 1Efficient networks 1Feature pyramids and necks 1Generative models 1Object detection 1Self-attention 1