Training data and decision boundary
Shading and the solid line follow the SGD weights live; the dashed line is LDA's fixed closed-form boundary.
RainNo rainSGD (live)LDA (closed-form)
Error vs. epoch
SGD: —LDA: —
Chapter 5 · Learning Decision Boundaries
SGD vs. LDA: learning a boundary one point at a time
2 features — cloud cover and humidity. LDA computes its straight-line boundary in one closed-form step from the class means and a shared covariance. Stochastic gradient descent learns a boundary the slow way: one training point at a time, one small correction at a time. Step through the arithmetic below and watch the two converge.
Dataset
Quality depends on the data too — pick a shape and watch the boundary and error curve change.
2 features · 2 classes
Calculation trace
Most recent completed update first — z, a, e and the resulting weights, one row per training point.
| # | cloud, humidity | y | z | a | e | w₁, w₂, b (after) |
|---|
w ← w + η(y − a(z))x — the only difference is whether a(z) is a hard 0/1 step or the smooth sigmoid. When a dataset isn't perfectly separable (the default Overlapping dataset caps LDA at 87.5% training accuracy), the perceptron can keep flipping weights on the same handful of overlapping points without ever fully stabilizing, while logistic regression keeps converging toward a stable minimum — switch to the Separable dataset above to watch the perceptron actually settle down too. LDA isn't learned iteratively at all: assuming both classes share one covariance, a Gaussian Bayesian classifier reduces to exactly this kind of straight line in closed form — see Chapter 5's Salmon or Seabass demo for the probabilistic view of the same idea.