Gradient Boosting is a machine learning technique that builds a strong model by combining many simple models (usually decision trees) in sequence, with each new one correcting the errors of the ones before it.
What it means in plain English
Unlike a random forest, which builds trees independently and averages them, gradient boosting builds trees one at a time, each focused on the mistakes the previous trees made. By repeatedly correcting errors, it gradually assembles a highly accurate model. Popular implementations are among the top performers for structured data and frequently win data-science competitions.
It can be very accurate but needs more careful tuning than a random forest.
A simple example
To predict credit risk, gradient boosting builds a first tree, sees where it went wrong, builds a second tree to fix those errors, and so on — each tree improving on the last until the combined model is highly accurate.
Why it matters
Gradient boosting is one of the most powerful methods for structured, tabular data, often delivering the best accuracy of any approach. It is a workhorse in industry for prediction tasks on business and financial data.
Related terms
- Random Forest — a different way of combining trees.
- Decision Tree — the building block used.
- Ensemble Learning — the broader principle.
Frequently asked questions
What is gradient boosting?
It builds an ensemble of models (often decision trees) sequentially, where each new model corrects the errors of the previous ones, producing a strong overall predictor.
Where does gradient boosting shine?
It is highly effective on structured/tabular data and powers popular libraries like XGBoost, LightGBM, and CatBoost, often winning data-science competitions.
