It took 3 weeks to understand the principle of xgboost algorithm

Algorithm process

First learn the decision tree , then learn the random forest , and finally come to xgboost. I thought it would be easier to transition so smoothly, but I didn't expect it to be so difficult. It took more than 3 weeks scattered and scattered, and after reading the explanations and elaborations of many articles, I barely got the algorithm principle of xgboost.

Among them, there are two reference sources, which I personally recommend: one is text and the other is video .

The algorithm flow chart of xgboost is directly given here, which is convenient for us to understand xgboost intuitively. For a training set, xgboost first uses CART tree training to obtain a model, which will generate a deviation value for each sample; then use the sample deviation value as a new training set, continue to use CART tree training to obtain a new model; repeat , until an exit condition is met.

The final xgboost model is the sum of all the above models. Suppose there are a total of M models, and the output of each model is defined as $f_i$, then the final output of the xgboost model ${\hat{y}}_i$为
$${\hat{y}}_i=\sum _{i=1}^M{f}_i(x_i)$$

Obviously, xgboost, like random forest, is also an integration of multiple models, but there are still many differences between them. To describe it in technical terms, xgboost is a boosting method whose core feature is to reduce deviation and the logic is serial; while random forest is a bagging method whose core feature is to reduce variance and the logic is parallel.

After figuring out the algorithm process, we also need to pay attention to the details in the process. There are two things that need to be understood: (1) What is a CART tree? (2) How to get the CART tree model?

CART tree

Simply understand, the CART tree is first a tree; on this basis, each leaf node $j$ will be assigned a node value$w_j$。

Assuming $i$ samples$x_i$and jjThe mapping relationship between$j$
$$j=q(x_i)$$
Take the following figure as an example:$q(x_1)=2,q\left(x_2\right)=1,q\left(x_3\right)=2$ .

then the$The\; node\; value\; of\; i$ samples is
$$w_{q(x_i)}$$
On the other hand, the jjSample set I j I_j$of\; j$ nodes$I_j$Can be expressed as
I j = { xi ∣ q ( xi ) = j } I_j= \{x_i|q(x_i)=j \}Ij​={ xi​∣q(xi​)=j }
Or the above picture as an example:I 1 = { 2 } , I 2 = { 1 , 3 } I_1=\{2\}, I_2=\{1,3\}I1​={ 2},I2​={ 1,3}。

From the concept of CART tree, we can easily find that two types of data are needed to determine a CART tree: tree structure and node value. Next, we first study clearly the optimal node value optimization strategy when the tree structure is fixed; on this basis, we return to try to determine the optimal tree structure.

best node value

When the tree structure is fixed, the node that each sample falls into is determined immediately. At this point, define the following objective function to measure the error of xgboost on the overall sample
$$\text{obj}=\sum _{i=1}^{n}l(y_i,{\hat{y}}_i^{(M)})+\sum _{m=1}^{M}Oh\left(f_m\right)$$
Among them, the meaning of the first item is the error of the sample itself,$n$ represents the sample size,$l(\cdot )$ is a function to measure the sample error, such as MSE, etc.,$y_i$for $The\; true\; value\; of\; i$ samples,${\hat{y}}_i^{\left(M\right)}$for $The\; predicted\; value\; of\; i$ samples; the second term is the regular term, which hasbeen used in linear model optimization: ridge regression and Lasso regression before. The main purpose is to reduce the risk of overfitting, and the goal here is also to reduce overfitting joint risk.

The second item is simple, first deal with
$$\sum _{m=1}^{M}Oh(f_m)=\sum _{m=1}^{M-1}Oh\left(f_m\right)+Oh\left(f_M\right)$$
From the algorithm flow, we can see that the CART tree is built one by one. When we want to optimize the$When\; there\; are\; M$ trees, the first$M-1$ tree has completed the calculation, that is, the regular term value of these trees has been determined, so we can ignore it when optimizing; for the$M$ trees,Ω \OmegaΩflow$\Omega$
$$Oh\left(f_M\right)=\gamma T+\frac{1}{2}l\sum _{j=1}^T{w}_j^2$$
Here, $\gamma$ and$\lambda$ is a penalty item,$T$ is the total number of leaf nodes of the tree. This definition can be understood as follows: On the one hand,$If\; the\; T$ value is large, it means that the depth of the tree is relatively deep, and the probability of overfitting will become higher, so use$\gamma$ punishes; on the other hand,$The\; w$ value is large, indicating that the tree will occupy a large proportion in the entire model, that is, the prediction result mainly depends on the tree, and the risk of overfitting will also increase at this time, so it is necessary to use λ \$\lambda$ is punished.

Now back to the first item. After having the specific expression of the second item, if given $l(\cdot )$ expression, it seems that the extremum can be obtained directly through gradient derivation. But in practice, this method is not feasible. This is mainly because$\hat{y}$It is obtained through the tree model, so the value is not continuous, so it is not derivable.

That being the case, we need other solutions. Fortunately, because the former $M-1$ CART tree has been determined, so you only need to pay attention to the$The\; node\; values\; of\; M$ trees are enough, so the first item can be transformed into
$$\sum _{i=1}^{n}l(y_i,{\hat{y}}_i^{(M)})=\sum _{j=1}^T\sum _{i\in I_j}l(y_i,{\hat{y}}_i^{\left(M\right)})$$
The value of this conversion is to convert the statistical logic of errors from the sum of samples to the sum of nodes, so that it can be compared with$\Omega$ uses the same variable, which facilitates combining operations of expressions.

To deal with this term, we will take a Taylor expansion on it and hold down to the second order term.

First review the Taylor formula
$$f(x+\delta x)=f(x)+f^{\prime }(x)\delta x+\frac{1}{2}{f}^{\prime \prime }(x)\delta x^2$$
Here put${\hat{y}}_i^{(M-1)}$defined as $x$ , then$\delta x$ is$w_j$，我们照葫芦画瓢进行泰勒展开
$$l(y_i,{\hat{y}}_i^{(M)})=l(y_i,{\hat{y}}_i^{(M-1)}+w_j)=l(y_i,{\hat{y}}_i^{(M-1)})+l^{\prime }(y_i,{\hat{y}}_i^{(M-1)})w_j+\frac{1}{2}{l}^{\prime \prime }(y_i,{\hat{y}}_i^{(M-1)})w_j^2$$
The first one is a constant and can be ignored. Let $g_i=l^{\prime }(y_i,{\hat{y}}_i^{(M-1)})$，$h_i=l^{\prime \prime }(y_i,{\hat{y}}_i^{(M-1)})$
。

The overall objective function becomes

$$\text{obj}=\sum _{j=1}^T\sum _{i\in I_j}[g_i{w}_j+\frac{1}{2}{h}_i{w}_j^2]+\gamma T+\frac{1}{2}l\sum _{j=1}^T{w}_j^2$$
due to $w_j$sum $i$无关，所以可以调整为
$$\text{obj}=\sum _{j=1}^T[w_j\sum _{i\in I_j}{g}_i+\frac{1}{2}{w}_j^2\sum _{i\in I_j}{h}_i]+\gamma T+\frac{1}{2}l\sum _{j=1}^T{w}_j^2$$
merge $w_j^2$项，得到
$$\text{obj}=\sum _{j=1}^T[w_j\sum _{i\in I_j}{g}_i+\frac{1}{2}{w}_j^2(l+\sum _{i\in I_j}{h}_i)]+\gamma T$$

令$G_j={\sum }_{i\in I_j}{g}_i$，$H_j={\sum }_{i\in I_j}{h}_i$, the above formula can be simplified as
$$\text{obj}=\sum _{j=1}^T[w_j{G}_j+\frac{1}{2}{w}_j^2(l+H_j)]+\gamma T$$
is a binary linear expression, the optimal solution is
$$w_j^{\ast }=-\frac{G_j}{l+H_j}$$
The corresponding optimal objective function value is
$${\text{obj}}^{\ast }=-\frac{1}{2}\sum _{j=1}^T\frac{G_j^2}{l+H_j}+\gamma T$$

Here also need to describe $g_i$sum $h_i$Specifically how it is calculated. For example, suppose we are optimizing the 11th CART tree, which means that the first 10 CART trees have been determined. These 10 trees pair samples ( $x_i,y_i=1$ ) the predicted value is$y_i=-1$ , assuming we are doing classification now, our loss function is
$$L\left(i\right)=\sum _i{y}_{i}ln(1+e^{-{\hat{y}}_i})+(1-y_i)ln(1+e^{{\hat{y}}_i})$$
because$y_i=1$ , the loss function becomes
$$L\left(i\right)=ln(1+e^{-{\hat{y}}_i})$$
to find the gradient, the result is
$$\frac{e^{-{\hat{y}}_i}}{1-e^{-{\hat{y}}_i}}$$
将${\hat{y}}_i=-1$ into the gradient expression, then$g_{11}=-0.27$。

Continue to derive the gradient expression, and after obtaining the second derivative expression, bring it into ${\hat{y}}_i$The value of $h_{11}$。

Understand $w_j^{\ast }$. Assuming there is only one sample at the node, then
$$w_j^{\ast }=(\frac{1}{l+h_j})(-g_j)$$
$-g_j$Represents the negative gradient direction, that is, the direction in which the value of the objective function drops the fastest, which is in line with our cognition; $\frac{1}{l+h_j}$Can be seen as the learning rate, if $h_j$A large value indicates that the gradient change is relatively large, that is, a small disturbance will bring about a great change in the objective function. At this time, the learning rate should be reduced, so it is also in line with our cognition.

In summary, as long as the tree structure is determined, the optimal value wi w_i of each node can be obtained by the above method$w_i$, so that the objective function value is minimized. Now, we are left with only the optimal tree structure.

optimal tree structure

In order to determine the optimal tree structure, this section introduces a commonly used method: the greedy algorithm.

As shown below. For the current node, there are three samples of A, B and C, and the optimal objective function value at this node can be calculated as
$${\text{obj}}_0=c-\frac{1}{2}\frac{(G_A+G_B+G_C{)}^2}{H_A+H_B+H_C+l}$$

Try to split the node, assuming that there may be three situations, [A, BC], [C, AB] and [B, AC], the corresponding optimal objective function values ​​are obj 1 = 2 γ − 1 2
$${\text{obj}}_1=2c-\frac{1}{2}\frac{G_A^2}{H_A+l}-\frac{1}{2}\frac{(G_B+G_C{)}^2}{H_B+H_C+l}$$
$${\text{obj}}_2=2c-\frac{1}{2}\frac{(G_A+G_B{)}^2}{H_A+H_B+l}-\frac{1}{2}\frac{(G_C{)}^2}{H_C+l}$$
$${\text{obj}}_3=2c-\frac{1}{2}\frac{(G_A+G_C{)}^2}{H_A+H_C+l}-\frac{1}{2}\frac{(G_B{)}^2}{H_B+l}$$

Calculate the change value of the objective function in the three cases, namely: ${\text{obj}}_1-{\text{obj}}_0$、${\text{obj}}_2-{\text{obj}}_0$和${\text{obj}}_3-{\text{obj}}_0$. Then take the split result corresponding to the largest change value as the best split method for the node next time.

Therefore, the process of determining the optimal tree structure is:
starting from the depth of the tree at 0:
(1) enumerate all available features for each leaf node;
(2) for each feature, the training samples belonging to the node according to the The feature values ​​are arranged in ascending order, and the best split point of the feature is determined through the above greedy logic, and the split benefit of the feature is recorded; (3) The feature with the
largest profit is selected as the split feature, and the best split point of the feature is used. The split point is used as the split position, and two new left and right leaf nodes are split on the node, and the corresponding sample set is associated with each new node; (4)
Go back to step 1 and repeat until the specific conditions are met until;

At this point, I finally figured out the algorithm principle of xgboost.