Explanation of GBDT algorithm principle and summary of commonly used training frameworks: XGBoost LightGBM CatBoost NGBoost

1 Basic knowledge points

1.1 Ensemble Learning

Ensemble Learning is the core concept in Machine Learning. Its main idea is summarized as follows: by training multiple weak learners to achieve the effect of a strong learner, the combined performance is better than that of any weak learner. Errors in machine learning can be roughly divided into two categories:One is the bias error (bias error): refers to the difference between the predicted value and the real value, and the other is the variance error (variance error): refers to the degree of dispersion of the predicted value as a random variable, and integrated learning can just reduce these problems. By combining the results of multiple classifiers, the deviation of model prediction can be reduced, especially for some unstable learners, so the learners learned by ensemble learning have higher stability. In ensemble learning, the common method isBaging和Boosting, and then we briefly describe these two methods.

1.2 Bagging and Boosting

To use Bagging or Boosting techniques, we must choose a base learner. For example, we can choose to classify trees, then Bagging and Boosting will be combined into a series of tree learners and become an integrated learner. Next, how do Bagging and Boosting train to get N learners?

• First:training data selection
  Each time, N new training data sets are generated from the original training data set, and then the N learners are trained separately. When generating each new training data set, Bagging randomly selects samples, which means that the probability of each sample appearing in the new training set is the same, and Boosting will select according to the weight of the sample, so some samples will be It has a higher probability of being selected in the new training set.

• second:training process
  The main difference between Bagging and Boosting is the training process.Among them, Bagging is parallel in the training phase, and each trainer is independent, while Boosting builds each trainer based on sequence. The establishment of a new trainer depends on the previous trainer, so it is not independent., the comparison chart is as follows:
  

In the Boosting algorithm, the selection of training data for each classifier depends on the prediction results of the previous classifier, so in each training step, the weight of the sample will be readjusted,The wrongly predicted data will increase the weight, and the probability will be more likely to be selected to enter the next classifier for training, focusing on the identification of these hard samples.

• third:forecasting process
  After training N learners, there is a difference between Bagging and Boosting in the prediction results. Bagging strategy, the final result is the average of the results of N learners, and the prediction result of Boosting is the weighted sum, expressed as follows:
  $$\text{Bagging}=\frac{1}{N}\sum _{i=1}^N{s}_i$$
  $$\text{Boosting}=\sum _{i=1}^N{w}_i{s}_i$$
  That middle weight $w_i$will be assigned according to the performance of each classifier's prediction,The better the performance of the learner, the greater the weight corresponding to. But it does not mean that Boosting must be better than Bagging. It needs to be considered according to multiple factors such as specific data sets and learners. IfIf the performance of a single learner is poor, it is difficult for Bagging to obtain a powerful learner, but the Boosting optimization strategy can just strengthen the effect of multiple learners. On the contrary, if every learner is overfitting, then Bagging is the best choice, and Boosting does not help to avoid overfitting。

1.3 Adaptive Boosting

Adaptive Boosting (AdaBoost) is a Boosting method. The core idea of ​​Boosting is to learn from the previous model error. andThe AdaBoost learning method mainly increases the weight of the misclassified samples, so that the next model pays more attention to the recognition effect of the misclassified samples.. The basic steps of training are as follows:

• train a tree model
• Calculate the error rate ee of this tree model error$e$
• Calculate the wieght of this decision tree based on the error rate:learning_rate * log((1-e) /e), so the larger the error rate e, the smaller the weight
• Update the weight of each sample: for samples that are paired by the model, the weight remains unchanged; for samples that are misclassified, the new weight is:old_weight * np.exp(weight of this tree), after updating, the weight of the sample becomes larger, in the next step, the identification of such misclassified samples will be strengthened
• Repeat the above steps until the trained tree has reached the maximum value
• Make the final prediction: predict each candidate set sample through a weighted voting mechanism

1.3 Gradient Boosting

Gradient Boosting is also a Boosting method. As we mentioned above, the core point of the boosting model is to learn from previous mistakes.And each iteration of Gradient Boosting directly fits the residual of the previous step (the partial derivative of the target loss function to the output value), so that the prediction result of the current t step is equal to the negative gradient direction of the target loss function to the prediction value of the previous step t-1 , so that each iteration ( $f_t(x_i)=f_{t-1}\left(x_i\right)-\frac{\partial L(y_i,f_{t-1}(x_i))}{\partial f_{t-1}\left(x_i\right)}$) continuously reduce the target loss loss, the algorithm flow is as follows:

$1.Initialize\_:f_0(x)={\text{argmin}}_{}{\sum }_{i=1}^{N}L(y_i,c)$

$2.\text{for}t=1\text{to}T:$

$\left(a\right)Calculatenegativegradient:{\hat{y}}_i=-\frac{\partial L(y_i,f_{t-1}(x_i))}{\partial f_{t-1}(x_i)},i=1,2,...N$

$\left(b\right)byminimizingthesquarederror,withthebaselearnerh{}_t\left(x\right)fit{\hat{y}}_i,$

$w_t={\text{argmin}}_w{\sum }_{i=1}^{N}L({\hat{y}}_i-h_t(x_i;w){]}^2$

$\left(c\right)Determinethestepsize\rho usingLinesearcharch\_\_\_\_{}_m,sothatLis\; the\; smallest$ ,

$r_t={\text{argmin}}_{}{\sum }_{i=1}^{N}L(y_i,f_{t-1}(x_i)+\rho h_t(x_i;w_t))$

$(d)f_t(x)=f_{t-1}\left(x\right)+r_t{h}_t(x;w_t)$

$3.outputf\_{}_M\left(x\right)$

2 GBDT algorithm

2.1 Principle

GBDT (Gradient Boosting Decision Tree) is a gradient boosting tree. Next, we will deduce the details of the algorithm in detail.

1) GBDT prediction result value expression
Suppose we have K trees, we use the function fk ( x ) f_k(x) for the predicted value of the kth tree$f_k\left(x\right)$ means, then for a sample$x_i$，最终预测的结果值为：
$$\widehat{y_i}=\sum _{k=1}^K{f}_k(x_i)$$

2）定义目标损失函数
损失函数用来度量模型预测的值与真实标签值之间的差异，同时一般会增加正则化项来惩罚模型的权重参数，避免模型过于复杂，但是基于树模型算法，由于没有权重参数需要优化，因此需要其它的方式来惩罚模型的复杂性。一般对于树模型，通常正则化项会从树的深度，树的叶子节点数或者叶子节点权重值的L2范数等作为因子。通常，如果一颗树的叶子节点越多，树越深越容易过拟合，或者叶子节点权重分值越高，这些都容易导致过拟合。综合这些问题，那么对于树模型，我们可以定义的目标损失函数如下：

$$Obj=\sum _{i=1}^{n}l(y_i,\widehat{y_i})+\sum _{k=1}^K\Omega (f_k)$$
其中$l(y_i,{\hat{y}}_i)$ measures the difference between the model prediction and the real label,$Oh\left(f_k\right)$ is a regularization term, which measures the complexity of the model and prevents the model from overfitting. Our goal is to minimize the above objective loss function.

3) Deformation of the target loss function
With the objective function, how does the model learn? Since what we want to train is a tree-based function $f_t\left(x\right)$ , not a numeric vector, cannot be solved by gradient descent. So another method called additive training (boosting) is needed to find the optimal solution.
Assuming that the initial value is 0, for each additional tree, the iterative form of the predicted result value is as follows:

${\widehat{y_i}}^{(0)}=0$
${\widehat{y_i}}^{(1)}=f_1(x_i)={\widehat{y_i}}^{(0)}+f_1(x_i)$
${\widehat{y_i}}^{(2)}=f_1\left(x_i\right)+f_2\left(x_i\right)={\widehat{y_i}}^{\left(1\right)}+f_2(x_i)$
…
${\widehat{y_i}}^{(t)}={\sum }_{k=1}^t{f}_k\left(x_i\right)={\widehat{y_i}}^{(t-1)}+f_t\left(x_i\right)$

It can be seen from the above formula that at step t, the final prediction result is the sum of all the results of the previous t-1 step and the result of the current tree (the learning rate factor is not considered here to control the weight of each tree) value scaling). We expand the objective function as follows:

$Ob{j}^{(t)}={\sum }_{i=1}^{n}l(y_i,{\widehat{y_i}}^{(t)})+{\sum }_{i=1}^{t}Oh(f_i)$
$={\sum }_{i=1}^{n}l(y_i,{\widehat{y_i}}^{(t-1)}+f_t(x_i))+Oh(f_t)+const$

Assuming our target loss function is the mean square error, the transformation is as follows:

$Ob{j}^{(t)}={\sum }_{i=1}^n(y_i-({\widehat{y_i}}^{(t-1)}+f_t(x_i)){)}^2+Oh(f_t)+const$
$={\sum }_{i=1}^n[2({\widehat{y_i}}^{(t-1)}-y_i)f_t(x_i)+f_t(x_i{)}^2]+Oh(f_t)+const$

Since the function $f_t(x_i)$与${\widehat{y_i}}^{(t-1)}$ has no connection. In the second step change of the above objective function, the formula omits the separate${\widehat{y_i}}^{(t-1)}$ term. The following is a solution to GBDT optimization based on the principle of Taylor's formula.

4) Taylor formula
Taylor's formula is one of the approximation methods often used to study the properties of complex functions, and it is also an important application content of functional differential calculus. If the function satisfies certain conditions, the Taylor formula can use the derivative values ​​of the function at a certain point as coefficients to construct a polynomial to approximate the function. The expansion of the formula is as follows:
$$f(x+\Delta x)\approx f(x)+f^{{}^{\prime }}(x)\Delta x+\frac{1}{2}{f}^{{}^{\prime \prime }}(x)\Delta x^2$$

5) The objective function is represented by Taylor expansion
Next, we carry out Taylor expansion on the objective function, and the solution process is as follows:
$$Ob{j}^{(t)}=\sum _{i=1}^{n}l(y_i,{\widehat{y_i}}^{(t-1)}+f_t(x_i))+Oh(f_t)+const$$
我们令:
$$g_i={\partial }_{{\widehat{y_i}}^{(t-1)}}l(y_i,{\hat{y}}^{(t-1)})$$
$$h_i={\partial }_{{\widehat{y_i}}^{(t-1)}}^{2}l(y_i,{\widehat{y_i}}^{(t-1)})$$
gi$g_i$sum $h_i$Represent the objective function $l(y_i,{\widehat{y_i}}^{(t-1)})$对${\hat{y}}^{}$The first and second derivatives of${}^{(}$${}^t$${}^{-}$${}^1$${}^{)\; .}$So our objective function Taylor expansion formula is:
$$Ob{j}^{(t)}\approx \sum _{i=1}^n\left[l\right(y_i,{\widehat{y_i}}^{(t-1)})+g_i{f}_t(x_i)+\frac{1}{2}{h}_i{f}_t^2(x_i)]+Oh\left(f_t\right)+const$$
If the target loss function is considered to be the mean square error, then$g_i$sum $h_i$计算结果为：
$$g_i={\partial }_{{\widehat{y_i}}^{(t-1)}}({\widehat{y_i}}^{(t-1)}-y_i{)}^2=2({\widehat{y_i}}^{(t-1)}-y_i)$$
$$h_i={\partial }_{{\widehat{y_i}}^{(t-1)}}^2({\widehat{y_i}}^{(t-1)}-y_i{)}^2=2So$$
we resolve the objective function as:
$$Ob{j}^{(t)}\approx \sum _{i=1}^n\left[l\right(y_i,{\widehat{y_i}}^{(t-1)})+g_i{f}_t(x_i)+\frac{1}{2}{h}_i{f}_t^2(x_i)]+Oh\left(f_t\right)+const$$
$$=\sum _{i=1}^n[2({\widehat{y_i}}^{(t-1)}-y_i)f_t(x_i)+f_t(x_i{)}^2]+Oh(f_t)+constBecause$$
the constant has no effect on the optimization of the objective function, the constant part is removed and further resolved as:
$$=\sum _{i=1}^n[g_i{f}_t(x_i)+\frac{1}{2}{h}_i{f}_t^2(x_i)]+Oh\left(f_t\right)$$
in that$g_i$sum $h_i$Represent the objective function $l(y_i,{\widehat{y_i}}^{(t-1)})$对${\hat{y}}^{}$The first and second derivatives of${}^{(}$${}^t$${}^{-}$${}^1$${}^{)\; .}$

6) Regularization term $Oh(f_t)$
Then the regularization term of the objective function $Oh\left(f_t\right)$ is how to express it? The first thing we need to make clear is that the purpose of the regularization term is to prevent the model from overfitting, so it is necessary to prevent the model from being too complex. Based on the tree model structure, we can define the regularization formula as follows: Ω ( ff ) = γ T + 1 2
$$Oh\left(f_f\right)=\gamma T+\frac{1}{2}l\sum _{j=1}^T{w}_j^2$$
where $T$ represents the number of leaf nodes,$w_j$Indicates the $The\; score\; of\; j$ leaf nodes, from the expression of the regularization formula, we can see that the regularization term prevents a tree from having too many leaf nodes and the value of the leaf nodes cannot be too large. As shown below:

则Ω \OmegaThe calculation result of$\Omega$$c^3+\frac{1}{2}l(4+0.01+1)$ , in the previous target loss function, we know$f_t\left(x\right)$ is the predicted score of the model result, then for the tree model structure we set:
$$f_t(x)=w_{q(x)}$$
其中$w\in {\text{R}}^T$ is a T-dimensional vector, and each value in the element represents the score of each leaf node. $q(x)$是映射函数，对于样本x，映射到树结构的具体哪个叶子节点。将正则化项和树模型结构表示带入上述目标函数为：
$$Ob{j}^{(t)}\approx \sum _{i=1}^n[g_i{f}_t(x_i)+\frac{1}{2}{h}_i{f}_t^2(x_i)]+\Omega (f_t)$$
$$=\sum _{i=1}^n[g_i{f}_t(x_i)+\frac{1}{2}{h}_i{f}_t^2(x_i)]+\gamma T+\frac{1}{2}\lambda \sum _{j=1}^T{w}_j^2$$
$$=\sum _{j=1}^T[(\sum _{i\in I_j}{g}_i)w_j+\frac{1}{2}(\sum _{i\in I_j}{h}_i+\lambda )w_j^2]+\gamma T$$

我们定义：
$$G_j=\sum _{i\in I_j}{g}_i$$
$$H_j=\sum _{i\in I_j}{h}_i$$
则我们对上述目标函数进一步化解得到：
$$Ob{j}^{(t)}=\sum _{j=1}^T[(\sum _{i\in I_j}{g}_i)w_j+\frac{1}{2}(\sum _{i\in I_j}{h}_i+\lambda )w_j^2]+\gamma T$$
$$=\sum _{j=1}^T[G_j{w}_j+\frac{1}{2}(H_j+\lambda )w_j^2]+\gamma T$$
Among them $w_j$Is the value we want to optimize, that is, the leaf node value in the tree structure, to minimize the above target loss function, we wj $w_j$take the derivative and set the result equal to 0：
$$\sum _{j=1}^T[G_j+(H_j+l)w_j]=0$$
, the following values ​​can be obtained:
$$w_j^{\ast }=-\frac{G_j}{H_j+l}$$
Will $w_j^{\ast }$value into the target loss function, the target loss function can be obtained as:
$$Ob{j}^{(t)}=-\frac{1}{2}\sum _{j=1}^T\frac{G_j^2}{H_j+l}+\gamma T$$

Get the final target loss function expression, what we want to solve is $g_i$sum $h_i$，where $G_j$and $H_j$Respectively represent the leaf node jjThe first derivativegg$of\; all\; samples\; of\; j$$sum\; of\; g$ fractions and second derivativehhsum of fractions of$h$. So for any kind of structure tree, we can calculate the target loss function value for the leaf node where the sample arrives, as shown in the following figure:

7) How to solve the optimal segmentation structure of a tree
In order to minimize Obj, we need to calculate all combined split trees, then calculate the target loss value of each tree according to the above formula, and finally select the tree structure with the smallest loss value. Is this possible? It is definitely not feasible, exhaustively enumerating all tree structures, the complexity is too high. So, do we need at each level of the treeDivide the node into left node and right node, depending on whether the target loss value after division is less than that before no division, and according to which feature point to split, split threshold, select the maximum gain (the target loss loss value is the lowest after the newly selected cut point), and the split gain gain is calculated as follows (loss before cutting - loss after cutting):
$$Gain=\frac{1}{2}[\frac{G_L^2}{H_L+l}+\frac{G_R^2}{H_R-l}-\frac{(G_L+G_R{)}^2}{H_L+H_R+l}]-\gamma So$$
whether the leaf node should be split, through the above formula, if the gain income is greater than 0, then you can continue to split, as for how to cut, you can$g$ and$h$ to sort, and then scan from left to right to select the optimal cutting point.

2.2 Training

After getting familiar with the principle of GBDT, let's take a look at how the GBDT model is trained, and what kind of model structure we get after training

training steps
Assuming we are using the mean square error loss function, the gbdt model training steps are summarized as follows:

• The first step (initial value determination): the initial value model prediction score of each sample is the average value of all samples: $f_0(x_i)=\frac{1}{n}{\sum }_{i=1}^n{y}_i$
• The second step (building the first tree): For each sample, according to the previous prediction results ${\widehat{y_i}}^0=f_0\left(x_i\right)$ value and the corresponding label value$y_i$, to get the objective function to the predicted value of the model in the previous step一阶导数$g_i=2({\widehat{y_i}}^0-y_i)$ and the second derivative$h_i=2$, and then traverse all feature types, and the range of feature values ​​corresponding to each feature. After splitting, calculate the Gain growth before and after splitting, select the splitting point with the largest growth, and then continue to split nodes down until before and after splitting The difference is lower than the set threshold, or the depth of the model, the number of leaf nodes, etc. exceed the set threshold. at last,The leaf node w value is calculated according to the above formula: $w=-\frac{G_j}{H_j+l}$, where $G_j，H_j$is the sum of the first-order derivatives and the sum of the second-order derivatives of all samples falling into this leaf node, so that a sample can be calculated, divided according to the rules, and which leaf node it falls intoFunction $f_1(x_i)$。
• The third step (building the second tree): For all samples, through all the previous trees, according to the segmentation conditions, the cumulative sum of the scores falling into the leaf nodes of each tree is the predicted score of the current sample: yi ${\widehat{y_i}}^1=f_0\left(x_i\right)+f_1\left(x_i\right)$ , according to the first tree building rule process, for the construction of the second tree, then calculate the score$f_2\left(x_i\right)$
• The fourth step (building the third tree): All samples go through the previously constructed tree and reach the leaf nodes according to the conditions, then the predicted result value of the sample at this time is: yi ^ 2 = f 0 ( xi ) + f 1 ( ${\widehat{y_i}}^2=f_0\left(x_i\right)+f_1\left(x_i\right)+f_2(x_i)$，根据最新结果，然后计算每个样本的$g_i,h_i$，选择增益最大的gain作为切分点。
• …
• 直到建树数量满足设置的阈值

模型结构
训练完后，就得到了N棵树，每一棵树的结构本质上是一连串的分段规则组成，根据输入样本的特征满足情况走树的不同分支，最后落入到树的某个叶子节点，其中落入到叶子节点的权重值就是这个样本在当前这课树的预测分值。

2.3 预测

当训练好了模型以后，预测的过程就简单了，假设有T棵树，则最终的模型预测结果为这个样本落入到每棵树的叶子节点分值之和，用公式表达如下：
$$\widehat{y_i}=\sum _{j=0}^T{f}_j(x_i)$$

3 训练框架

接下来介绍优化Gradient Boosting算法的几种分布式训练框架，这些框架支持分布式训练，树的调优，缺失值处理，正则化等避免过拟合问题。

3.1 XGBoost

XGBoost: A Scalable Tree Boosting System 是由2014年5月，由DMLC开发出来的，目前是比较受欢迎，高效分布式训练Gradient Boosted Trees算法框架，包含的详细资料可以参考官方文档: 官网文档。

3.2 LightGBM

LightGBM: A Highly Efficient Gradient Boosting Decision Tree 是由Microsoft Teams in January 2017Aiming at some problems existing in the XGBoost framework, a more efficient learning framework is designed, mainly based onGradient unilateral sampling GOSS ((Gradient Based One Side Sampling) and mutually exclusive feature merging EFB(Exclusive Feature Bundling) to speed up the learning efficiency of the model. For details, please refer to the official document official website document . The following is a simple code for rank sorting based on lightgbm:

import lightgbm as lgb
import numpy as np 
import pandas as pd
from sklearn.model_selection import train_test_split
import seaborn as sns
import matplotlib.pyplot as plt
import seaborn as sns
import shap
import graphviz

#读取数据，显示前面20行
df = pd.read_csv('train.csv')
df.head(20)
#显示数据列名称
df.columns
#抽取x,y对应的字段
X = df.drop(['label','query','term'], axis=1)
y = df.label
group=np.loadtxt('./group.txt')
#训练数据
train_data = lgb.Dataset(X, label=y, group=group,free_raw_data=False)
#参数定义
params = {
    
    
    'task' : 'train', 
    'boosting_type': 'gbdt',
    'objective': 'lambdarank',
    'num_iterations': 200,
    'learning_rate':0.1,
    'num_leaves': 31,
    'tree_learner': 'serial',
    'max_depth': 6,
    'metric': 'ndcg',
    'metric_freq': 10,
    'train_metric':True,
    'ndcg_at':[2],
    'max_bin':255,
    'max_position': 20,
    'verbose':0
}
#指明类别特征
categorical_feature=[0,1]
#训练
gbm=lgb.train(params,
              train_data,
              valid_sets=train_data,
              categorical_feature=categorical_feature)
#模型保存
gbm.save_model('model_large.md')
#预测
bst = lgb.Booster(model_file='model_large.md')
df_test = pd.read_csv('test.csv')
y_pred = bst.predict(test)

#feature重要度
fea_imp = pd.DataFrame({
    
    'imp': bst.feature_importance(importance_type='split'), 'col': X.columns})
fea_imp = fea_imp.sort_values(['imp', 'col'], ascending=[True, False]).iloc[-30:]
fea_imp.plot(kind='barh', x='col', y='imp', figsize=(10, 7), legend=None)
plt.title('Feature Importance')
plt.ylabel('Features')
plt.xlabel('Importance');
# 基于shap特征分析
explainer = shap.TreeExplainer(bst)
shap_values = explainer.shap_values(X)
shap.summary_plot(shap_values, X, plot_type="bar")
shap.summary_plot(shap_values, X)
shap.dependence_plot('entropy', shap_values, X, interaction_index=None, show=True)

3.3 Cat Boost

CatBoost: unbiased boosting with categorical features 是由In April 2017, Russian search giant Yandex developedA framework for optimizing xgboost is developed. The biggest advantage of this framework is that it can handle categorical features. Compared with LightGBM, it does not require label encoding for categorical features, which is convenient for users to quickly operate. Detailed official website: official website website , the following is a detailed comparison chart of the three:
training loss comparison:

model performance comparison: the left is the CPU machine, and the right is the GPU machine.

It can be seen that Catboost has a certain improvement in the index convergence effect compared with XGBoost and LightGBM Under the condition of , the performance of the model has been greatly improved. The following is a simple code example for training based on Catboost:

import numpy as np 
import pandas as pd
import os
from sklearn.metrics import mean_squared_error
from sklearn import feature_selection
from catboost import CatBoostClassifier
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
import seaborn as sns
import matplotlib.pyplot as plt

#读取数据
df = pd.read_csv('data.csv')
df.head()
#显示特征名
df.columns
#显示某个特征名的数据情况
pd.set_option('display.float_format', '{:.2f}'.format)
df.f_ctf.describe()
#特征分布显示
plt.figure(figsize = (10, 4))
plt.scatter(range(df.shape[0]), np.sort(df['f_ctf'].values))
plt.xlabel('index')
plt.ylabel('f_ctf')
plt.title("f_ctf Distribution")
plt.show();
#训练样本划分
X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.20, random_state=42)
#指明categorical特征
categorical_features_indices=[1,2,3]
#模型训练
model =  CatBoostClassifier(iterations=700,
                             learning_rate=0.01,
                             depth=15,
                             eval_metric='AUC',
                             random_seed = 42,
                             bagging_temperature = 0.2,
                             od_type='Iter',
                             metric_period = 75,
                             loss_function='Logloss',
                             od_wait=100)
model.fit(X_train, y_train,
                 eval_set=(X_valid, y_valid),
                 cat_features=categorical_features_indices,
                 use_best_model=True,
                 plot=True)

#特征重要度显示
fea_imp = pd.DataFrame({
    
    'imp': model.feature_importances_, 'col': X.columns})
fea_imp = fea_imp.sort_values(['imp', 'col'], ascending=[True,False]).iloc[-30:]
fea_imp.plot(kind='barh', x='col', y='imp', figsize=(10, 7), legend=None)
plt.title('CatBoost - Feature Importance')
plt.ylabel('Features')
plt.xlabel('Importance');

3.4 NGBoost

NGBoost: Natural Gradient Boosting for Probabilistic Prediction is a relatively new training Gradient Boosting algorithm framework. AtOctober 2019 by Andrew Ng's team at StanfordPublished. The github code is recorded in: NGBoost Github , the core point isIt uses natural gradient boosting, a modular boosting algorithm for probabilistic prediction. The algorithm consists of a ​base learner​, a ​parameter probability distribution​ and a ​scoring rule​.

4 Combination of tree model and depth model

The tree model has strong interpretability and stability, but the disadvantage is that it has no semantic features and the generalization ability is not strong enough. Therefore, in actual scenarios, the tree model can be combined with the deep learning model. For example, a simple average may have It must be improved. This is a simple experiment to compare the effect. The bert model has a certain improvement in the emotion classification task compared with the gbdt model. However, combining the results of bert and gbdt and simply calculating an average value has the best effect. For details, please refer to the one made by someone. Simple comparison test: bert vs catboost

6 References

1 GBDT Youtube Video