XGBoost/LightGBM/CatBoost (briefly)
Sunday October 3, 2021
There are many explainers of the popular gradient boosted tree models, but this is short.
| XGBoost | LightGBM | CatBoost |
|---|---|---|
| search missing high and low | search, then assign missing | specify missing high or low |
| "normal" balanced trees | leaf-first tree growth | oblivious trees (tables) |
| you handle categories | smart categorical ordering | permuted target coding |
| weighted quantile sketch | sample high-grad examples | permuted boosting |
| regularized objective | exclusive feature bundling | learns category interactions |
| 2016 paper | 2017 paper | 2017 paper |
This is close to correct, I think. It probably won't help you understand what's going on, but if you already know it might help jog your memory. The models all work pretty well.