Gradient boosted decision trees

XGBoost: improve the guess, one round at a time

Each round fits one small tree per class on the residual—the gap between the true label and the current softmax probability—and adds a gentle correction. Two features and up to three classes, so every round needs one tree per class.

Training data and decision surface

Start with equal scores for every class.

2 features · 2 classes

Weekly practice time and quiz score. Click a point to inspect its class scores; click empty space to test a new point.

One tree per class, this round

What did this round actually add?

Each leaf contributes a small adjustment to that class's score, scaled by η = 0.50. The note under each tree says whether the split moved or just its values changed.

This is the core boosting idea. Real XGBoost also uses second-order gradients, regularization, row/column sampling, and other safeguards. Here each round's trees are grown greedily from the visible data so the sequence stays easy to follow.