Chapter 5 · Learning Decision Boundaries
Activation functions: why not just a step?
Same rain-day generator and the same shared update rule as SGD vs. LDA — all six activations train at once, under the identical dataset, steepness k, learning rate η, and sample order. Every difference in the race below comes from the activation function alone.
Dataset
Quality depends on the data too — pick a shape and watch the race and table change.
The race: error vs. epoch
Same w=0 start, same sample order each epoch — the only thing that differs is a(z).
"Blocked" = points where |a′(z)| < 0.01 right now — why a line is stuck, not just that it is.
| Activation | Peak | Blocked | Acc. | Status |
|---|
w ← w + η·k·e·a′(z)·x — the error e only moves the weights if the slope a′(z) lets it. For Step, a′(z) = 0 everywhere it's defined, so Δw = 0 no matter how wrong the guess is: watch its line in the race sit dead flat forever (at 50% on a balanced dataset — try Imbalanced to see the flat line land somewhere else instead), and its "Gradient-blocked" count sit at 24/24 below. As Sigmoid/Tanh/Arctan get steeper (higher k), two different failure modes show up: either the peak slope right at the boundary overshoots on every update and the error never settles ("Oscillating"), or one early big swing pushes most points into saturation, freezing training far from a good boundary despite a flat, "stable-looking" line ("Stuck (saturated)"). ReLU sits in between: a clean slope of 1 on one side, a dead 0 on the other — the same dying-ReLU problem seen in real networks. This is the single-unit version of the vanishing/exploding-gradient problem multi-layer networks face at every layer — see this book's neural-network chapter for the deep-network version.