Topic · Mechanisms

Activation functions

Everything on this site about activation functions, in the order to read it: a page is listed after the pages it builds on. The figures below are the ones those pages use to explain it; each is live: move a slider and it recomputes.

In reading order

  1. Paper · read first: YOLOv4 — Version 1: Bag of Freebies

The figures these pages use

Pulled from the pages above. Move a slider and the figure recomputes, exactly as on its page.

From YOLOv4 — Version 2: Bag of Specials › 1. Mish Activation

-1.0
Graph of the ReLU function. ReLU is a ramp that starts at 0. Left of 0 both the output and the slope are zero.

From YOLOv4 — Version 2: Bag of Specials › 1. Mish Activation

-1.2
Graph of the Mish function. Mish dips a little below zero for negative inputs instead of cutting them off, then follows the input.

From YOLOv4 — Version 2: Bag of Specials › 1. Mish Activation

-1.0
Mish against ReLU and Swish. Mish is smooth everywhere and lets small negative values through instead of cutting them to zero.

From YOLOv4 — Version 2: Bag of Specials › 2. CSP(Cross Stage Partial Connections)

166
What each dense layer reads. Input channels seen by each layer of a dense block. The same early channels (and their gradients) are reused by every later layer, which is the duplication CSP removes.

From YOLOv4 — Version 2: Bag of Specials › 6. DIoU NMS⁹

140.5
Keep or suppress, by DIoU. The dashed box M is the one already kept. Slide the candidate B_i across it: ordinary NMS looks only at IoU, DIoU-NMS also asks how far apart the centres are.

Tags that go with it

Attention modules 1Feature pyramids and necks 1Object detection 1