XGBoost/LightGBM/CatBoost (briefly)

Sunday October 3, 2021

There are many explainers of the popular gradient boosted tree models, but this is short.

XGBoost LightGBM CatBoost
search missing high and low search, then assign missing specify missing high or low
"normal" balanced trees leaf-first tree growth oblivious trees (tables)
you handle categories smart categorical ordering permuted target coding
weighted quantile sketch sample high-grad examples permuted boosting
regularized objective exclusive feature bundling learns category interactions
2016 paper 2017 paper 2017 paper

This is close to correct, I think. It probably won't help you understand what's going on, but if you already know it might help jog your memory. The models all work pretty well.