Understanding Attention Modules: CBAM and BAM — A Quick Read

In this article, we will be going through two articles quickly viz. Bottleneck Attention Modules(BAM)¹ and Convolutional Block Attention Modules(CBAM)².

Recently, many different SOTA networks have leveraged these attention mechanisms that have significantly improved and refined real-time results.

Lightweight network and straightforward implementations have made it easier to incorporate directly into the feature extraction part of convolutional neural networks.

So, let’s get started by paying some ATTENTION to these topics.

PS: Stay tuned for exciting new articles like this in future by following VisionWizard Page.

1. Attention Module: What is?

• Attention modules are used to make CNN learn and focus more on the important information, rather than learning non-useful background information. In the case of object detection, useful information is the objects or target class crop that we want to classify and localize in an image.
• The attention module consists of a simple 2D-convolutional layer, MLP(in the case of channel attention), and sigmoid function at the end to generate a mask of the input feature map.

• It takes a CxHxW feature map as an input and gives 1xHxW(or CxHxW in case of 3D Attention Map) as an output attention map. This attention map is then multiplied element-wise with the input feature map to get a more refined and highlighted output.
• Generally, attention mechanisms are applied to spatial and channel dimensions. These two attention mechanisms viz. Spatial and Channel Attention Map generation is done sequentially as shown in Fig. 2 or parallelly.
• These attention mechanisms first experimented in residual architectures. The below-given figure is an overall structure of an attention layer.

2. Bottleneck Attention Modules(BAM)

• As we discussed, the attention module as a whole consists of the Channel Attention Module and Spatial Attention Module. For the given input feature map F ∈ C×H×W, BAM infers a 3D attention map M(F) ∈ C×H×W. The refined feature map F’ is computed as:-

• BAM is residual: the attention map M(F), with values between 0 and 1, is multiplied into the feature F and the result is added back to F.
• Take one position with F = 0.8. With M = 1 it becomes 0.8 + 0.8 × 1 = 1.6, doubled. With M = 0 it is 0.8 + 0 = 0.8, untouched. So attention can only emphasise, never erase, and the original gradient path through F stays intact.
• The figure below shows a feature map, an attention map that favours the right-hand blob, and the result. Turn the strength down to zero and F' becomes F.

attention strength

1.0

• Here ⊗ denotes element-wise multiplication. They adopt a residual learning scheme along with the attention mechanism to facilitate the gradient flow. So after multiplying the attention mask, they again add the output of with the input tensor F. To design an efficient module, they first compute the channel attention Mc(F) ∈ C×1×1 and the spatial attention Ms(F) ∈ 1×H×W at two separate branches, then compute the attention map M(F) as shown in above Fig. 3. Here σ is a sigmoid function. Both branch outputs are resized to C×H×W before addition.

• The two branches produce raw scores of different shapes: channel M_c(F) is C×1×1 and spatial M_s(F) is 1×H×W. They are broadcast to C×H×W, added, and squashed by a sigmoid so the result is a gate between 0 and 1.
• Adding before the sigmoid lets either branch open a position on its own. With M_c = 1 and M_s = −2 the gate is σ(−1) = 0.27; push M_c up to 4 and it is σ(2) = 0.88, even though the spatial branch has not changed its mind.
• Move the two scores in the figure below and see how the gate responds to each branch alone and to both together.

M_c (channel score)

1.0

M_s (spatial score)

-2.0

2.1 Channel Attention Module

• Steps to generate channel attention map are:-

1. Do Global Average Pooling of feature map F and get a channel vector Fc∈ Cx1x1.
2. Pass this Fc to a small MLP of one hidden layer of dimension C/r. Here r is a reduction ratio for the hidden channel(For example if the channel vector length is 1024 and reduction ratio r is 16, then the number of neurons in the hidden layer will be 64).
3. Add Batch Normalization Layer in front of this MLP.

• The channel branch in three steps: global average pooling turns each channel into one number, a two-layer MLP squeezes the C numbers down to C/r and back up to C, and batch normalisation follows. W₀ and W₁ are the two weight matrices.
• r is the reduction ratio that keeps the module cheap. For C = 1024 and r = 16 the hidden layer has 64 neurons, so W₀ and W₁ together hold 2 × 1024 × 64 = 131,072 weights; a 3×3 convolution of the same width holds 1024 × 1024 × 9 = 9.4M, so the branch adds about 1.4% on top.
• Change the number of channels in the bar chart below and see how the size of the branch depends on r.

channels

512

2.2 Spatial Attention Module

• Steps to generate spatial attention map are:-

1. Input feature map F is passed through chains of 1x1 and 3x3 Convolutional Layers. Here 3x3 convolutional layers contain a dilation value d=4. Dilation value is used to increase the effective receptive field of the network. We can see from Fig. 4 the effect of increased dilation value. Higher the value of dilation more will be the receptive field.
2. The first 1x1 layer reduces the channel dimensions to C/r. This reduced feature map is passed to two 3x3 convolutional blocks with d=4. Finally, output channels of these layers are reduced to 1xHxW by the last 1x1 convolutional layer followed by Batch Norm as shown in Fig. 3 to generate Ms(F).

• The spatial branch: a 1×1 convolution reduces the channels to C/r, two 3×3 convolutions with dilation d gather context, and a final 1×1 convolution produces one channel.
• Dilation is what lets a 3×3 kernel see a wide neighbourhood without extra weights. One 3×3 convolution with dilation d covers 2d + 1 pixels, and n of them in a row cover 2nd + 1. BAM uses d = 4 and two of them, so each output pixel looks at a 2 × 2 × 4 + 1 = 17 × 17 window instead of 5 × 5, still with 9 weights per channel pair.
• The figure below shows the window one output pixel sees for each dilation value; change the number of 3×3 convolutions in a row and watch the window grow.

3×3 convs in a row

2

2.3 Combination of Channel and Spatial Attention Maps

• Once we get Mc and Ms, they are added elementwise to get the final attention map. They chose addition over multiplication due to the smooth gradient flow at the time of backpropagation.
• As we can see that both will be having different dimensions(Cx1x1 and 1xHxW), so feature map is broadcasted in spatial and channel dimensions respectively to generate CxHxW.

2.4 What to do with this generated Attention Maps?

• This attention map M(F) is multiplied with the input feature map and finally added again to get a more refined and highlighted map. This is the structure followed by them and resembles the ResNet family.

2.5 Where to keep BAM?

• This module is kept at every bottleneck of ResNet architecture. BAM denoises low-level features such as background texture feature at the early stage and then gradually focuses on the exact target which is a high-level semanticity.

3. Convolutional Block Attention Module(CBAM)

• Given an intermediate feature map F ∈ C×H×W as input, CBAM sequentially infers a 1D channel attention map Mc ∈ C×1×1 and a 2D spatial attention map Ms ∈ 1×H×W as shown in Fig. 6. The overall attention process can be summarized as:

• CBAM applies its two gates one after the other rather than adding them. First M_c scales each channel of F to give F′, then M_s scales each spatial position of F′ to give F″.
• Because the gates multiply, a feature survives only if both its channel and its position are judged useful: a channel gate of 0.9 and a spatial gate of 0.3 leave 0.9 × 0.3 = 0.27 of the value, and a gate of 0 on either side wipes it out.
• In the figure below rows are channels and columns are positions. Sharpen the two gates and watch whole rows dim after M_c and whole columns dim after M_s.

channel gate sharpness

0.8

spatial gate sharpness

0.8

• Here ⊗ denotes element-wise multiplication. During multiplication, the attention values are broadcasted (copied) accordingly: channel attention values are broadcasted along the spatial dimension, and vice versa. F’’ is the final refined output. Fig. 6 depicts the computation process of each attention map.

3.1 Channel Attention Map

• Here Channel Attention Map follows the same generation process as BAM, but here with Average Pooling, Max Pooling is also added for getting more distinctive channel features as shown in Fig. 6.

• CBAM's channel attention pools each channel in two ways, by average and by maximum, sends both C-vectors through the same shared MLP (W₀ then W₁, with the reduction ratio r), adds the two results and applies a sigmoid.
• Why both? Average pooling measures how active a channel is overall. Max pooling notices a channel that fires strongly in only one small place, which the average would miss: one spike of 0.9 among 100 otherwise silent positions averages to 0.9/100 = 0.009, while its max is still 0.9.
• The figure below pools eight channels both ways. Channel 6 is silent except for one spike: move the spike height and its average stays near zero while its max follows the spike.

height of the spike in channel 6

0.9

• Here σ denotes the sigmoid function, W0 ∈ C/r×C, and W1 ∈ C×C/r. Note that the MLP weights, W0 and W1, are shared for both inputs and the ReLU activation function is followed by W0.

3.2 Spatial Attention Map

• Spatial map generation steps:-

1. Take the input feature map F and generate two intermediate feature maps viz. Fsavg, and Fsmax ∈ 1xHxW.
2. Concatenate these two outputs GAP(Global Average Pool)and MP(Max Pooling)and pass it through a small convolutional block of 7x7 kernel size. Here unlike BAM to increase receptive field, CBAM uses large kernel sizes to accomplish the same. Also, this is a simple convolutional block with d=1.

• Spatial attention in CBAM pools across the channel axis instead, giving two single-channel maps: the average and the max at every position. They are stacked, a 7×7 convolution turns the pair into one map, and a sigmoid makes it a gate.
• The large kernel lets each position's weight depend on its surroundings: with k = 7 the gate at a position is built from 7 × 7 × 2 = 98 numbers, with k = 1 from only the 2 at that position itself.
• The figure below shows the two pooled maps and the gate a k×k filter builds from them (a box filter stands in for the learned one). Increase k: a bigger kernel smooths the gate and widens the region it opens.

kernel size k

7

3.3 How to leverage this generated attention maps

• As shown in Fig. 7, these generated feature maps are arranged sequentially. First, the Channel attention map is applied on the input map, followed by the application of the spatial map. Finally, this F’’ is added to the previous input convolutional layer — ResNet Family(This is the shortcut layer for that particular ResNet block).

3.4 Where to keep CBAM

• Unlike BAM, these modules are placed inside residual blocks as well as at the bottlenecks.

3.5 What is the difference between CBAM and BAM?

In the case of BAM, only GAP was used to get the statistics of the feature map in spatial and channel dimensions. Whereas, CBAM also considered to use the MaxPooling with Average Pool. They proved that using Maxpooling accounts to generate the most salient features from the feature map and compensate the GAP output which encodes the global statistics softly.

In the case of BAM, the convolutional operation was done using dilation value d=4 to increase the receptive field as we go deep in the network. Whereas, CBAM used the higher kernel size of 7x7 and normal convolutional layer with d=1 to incorporate the same.

In the case of BAM, parallel generation of spatial and channel attention maps were taken into consideration which was later added to get the final attention map. Whereas in the case of CBAM, a sequential approach was used. Firstly, the channel attention map was calculated and then the Spatial map was finally obtained from the generated intermediate feature map(Channel Map×Input Feature Map).

Order of the sequential arrangement in the case of CBAM is Channel Attention → Spatial Attention.

Official Pytorch Implementation of BAM and CBAM: [code]

If you have managed to reach here, then I believe you are a part of an elite group who have a thorough understanding on attention modules — CBAM and BAM.

Please feel free to share your thoughts and ideas in the comment section below.

If you think that article was helpful, please do share it, and also clap(s) would hurt no one.

4. References

[1] Bottleneck Attention Module

[2] Convolutional Block Attention Module