Chapter 5 · Learning Decision Boundaries

Maximum margin: training a linear SVM one point at a time

Same 2 features, same rain days as SGD vs. LDA — cloud cover and humidity. This time the classifier doesn't just want to land on the correct side of the boundary; it wants a wide, empty buffer around it. Only the points inside that buffer — the support vectors — ever push the boundary. Step through the arithmetic and watch which points are outlined.

Dataset

The exact same three rain-day sets as SGD vs. LDA — pick a shape and watch the margin and support vectors change.

Configure
0.02

Smaller λ → narrower margin, hugs the training points closely. Larger λ → wider margin, more tolerant of violations. Picking a dataset or dragging λ retrains from scratch so the margin updates live — press Reset to rewind to w=0 and walk the arithmetic step by step instead.

Epoch 1 · Sample 1 / 24 · Step 0 / 8

Training data, margin, and support vectors

The solid line is the SVM boundary; the two thin dashed lines flanking it are the margin (z = ±1). Outlined points are support vectors — the only points with any pull on the current weights.

2 features · 2 classes
RainNo rainSVM boundaryMargin (z=±1)LDA (closed-form)

Error vs. epoch

SVM: —LDA: —
Margin width 2/‖w‖—
Support vectors—

Calculation trace

Most recent completed update first — z, margin, and whether the point was a support vector (SV) this round.

#cloud, humidityyzmarginSV?w₁, w₂, b (after)
Same shrink-and-nudge structure as gradient descent, with a margin instead of a residual. Every update shrinks w a little toward zero (the λw term — this is what widens the margin), and only the points that are inside the margin or misclassified (margin = y·z < 1) also get to pull the boundary toward them. That's why most points end each round with no visible effect: once a point is safely past the margin, it stops mattering, no matter how far past it is. Raise λ and watch the margin widen and the support-vector count shrink to just the closest few points on each side; lower it toward 0 and the boundary chases every training point, margin included, the way SGD's perceptron does. LDA's dashed line is the same fixed closed-form reference used across Chapter 5 — see Salmon or Seabass for its probabilistic derivation.