Super detailed! Introduction to XGBoost principle

This article was written by Yuan Zhecheng himself. Don’t count it as a plagiarism check. Thank you! XGBoost is the earliest of the three most popular GBDT models. The latter two are improved on the basis of XGBoost. Therefore, this article introduces XGBoost to have a certain understanding of the GBDT model.

gradient boosted tree

Improved integration algorithm

Build multiple weak estimators on the data and aggregate the modeling results of all weak estimators to obtain better regression or classification performance than a single model. There are many ways to integrate different weak evaluators, including random forests that resume multiple parallel weak evaluators at once, and gradient boosting trees that build weak evaluators one by one. Gradient boosting trees can be either regression trees or classification trees. The following uses gradient boosting regression trees as examples.
First, the modeling process of gradient boosting regression trees is roughly as follows: initially build a tree, and then gradually iterate, adding a tree during each iteration, and then gradually form strong estimators of many tree models.

For decision trees, each sample that is put into the model $i$ will eventually fall to a leaf node. For ordinary regression trees, the value on each leaf node is the mean of all samples on this leaf node.
For gradient boosting regression trees, the prediction result of each sample is the weighted sum of the results on all trees:
$${\hat{y}}_i=\sum _{k=1}^K{c}_k{h}_k\left(x_i\right)$$
where$K$ is the total number of trees,$k$ represents the$k$ trees,$c_k$is the weight of this tree, $h_k$Represents the prediction result on this tree (i.e., the average of all samples).
One of the optimizations of XGBoost is to optimize the prediction results. It is no longer the weighted sum of the average values ​​of all samples, but introduces a prediction score, also called leaf weight.
$${\hat{y}}_i=\sum _{k=1}^K{f}_k\left(x_i\right)$$
in which$f_k\left(x_i\right)$ is the prediction score, and the other letters have the same meaning as before.

Parameters:n-estimators

Use this parameter to determine how many trees there are in total.
First of all, the number of trees in XGB determines the learning ability of the model. The greater the number of trees, the stronger the learning ability of the model. As long as the number of trees in XGB is sufficient, even if there is only a small amount of data, the model can learn 100% of the information of the training data, so XGB is also a model that is inherently overfitting. But in this case, the model becomes very unstable.
Second, when the number of trees in XGB is small, the impact on the model is greater. When the number of trees is already large, the impact on the model is relatively small and there can only be weak changes. When the data itself is overfitting, using too many trees will have little effect and will instead waste computing resources.
Third, increasing the number of trees has a limit to its impact on the model. Initially, the performance of the model will increase with the number of trees in XGB, but
after more trees there are, the more effective the model will be. Decline, which also shows that violently increasing n_estimators may not be effective.

Parameterssubsample

Before we train the model, there must be a huge data set. We all know that tree models are inherently overfitting models, and if the amount of data is too large, the calculation of the tree model will be very slow. Therefore, we need to perform bootstrap sampling (bootsrap) on our original data set. Sampling with replacement prevents overfitting.
In the gradient boosting tree, we need to build a new tree every iteration, so we need to extract a new training sample with replacement in each iteration. However, this does not guarantee that every time a new tree is created, the integration effect will be better than before. Therefore, we stipulate that in the gradient boosting tree, each time an estimator is constructed, the model will be more focused on those samples in the data set that are prone to be misjudged. (Similar to adding priority to sampling in DQN.)
First, we have a huge data set. When building the first tree, we perform initial and replacement sampling on the data, and then model. After the modeling is completed, we conduct an evaluation of the model, and then feed back the samples with incorrect predictions by the model to our data set, and one iteration is completed. Next, we need to build a second decision tree, so we start sampling with replacement for the second time. But this sampling with replacement is different from the first random sampling with replacement. In this sampling, we increased the weight of samples that were incorrectly judged by the first tree. In other words, samples judged incorrectly by the first tree are more likely to be selected by us. Modeling based on this weighted training set, our new decision tree will be more inclined to these samples with larger weights that are easily judged incorrectly. After the modeling is completed, we feed back the incorrectly judged samples to the original data set. In the next iteration, the weight of the samples that were judged wrong will be greater, and the new model will be more inclined to these samples that are difficult to judge. This iterates over and over again.
Generally speaking, this parameter does not have much impact on the results and is generally left at the default value.

Reference number: eta( $the$ )

From a data perspective, we make the model more inclined to work hard on samples that are difficult to judge. However, it does not mean that as long as I build a new decision tree that favors difficult samples, it will be able to help me judge the difficult samples correctly. Difficult samples are given increased weight because the previous tree failed to judge it correctly, so for the next tree, the difficulty of the test set it has to judge is higher than the difficulty of the data encountered by the previous tree. Yes, it will become increasingly difficult to judge all these samples correctly. What if the newly created tree is not as good as the previous tree in judging difficult samples? Therefore, in addition to ensuring that the model gradually tends to the direction of difficult samples, we must also control the generation of new weak classifiers. We must ensure that, Each time a new tree is added, it must be the tree with the best prediction effect for this new data set.
A similar method to gradient descent is used here:

now we want to solve the optimal result of the ensemble algorithm, then we should be able to use the same idea: we first find a loss function, which should be able to be obtained by bringing in our prediction results Measure the prediction performance of our gradient boosted tree on the sample. We then use gradient descent to iterate our ensemble algorithm:
$${\widehat{y_i}}^{(k+1)}={\widehat{y_i}}^{(k)}+f_{k+1}\left(x_i\right)$$ We let this process continue to iterate until we find y ^ \hat{y}
that can minimize the loss function$\hat{y}$. A parameter eta \eta
is also used in XGB$\eta$ to control the iteration rate, so that our update method becomes as follows:
$${\hat{y}}_i^{(k+1)}={\hat{y}}_i^{(k)}+\eta f_{k+1}\left(x_i\right)$$
$The\; larger\; \eta$ is, the faster the algorithm converges, and it may not be able to converge to the optimum;$The\; smaller\; \eta$ is, the more accurate solution can be found, but the algorithm converges slowly.
Generally speaking$eta$ can be fine-tuned as follows, roughly between [0.01, 0.2].

Gradient Boosting Algorithm Summary

The gradient boosting algorithm mainly consists of 3 parts:
1. A loss function that can measure the effect of the integration algorithm and can be optimized;
2. A weak estimator $f_k\left(x\right)$ ;
3. A means to integrate weak evaluators, such as iteration means, sampling means, sample weighting, etc.
XGBoost operates on these three core elements of the gradient boosting tree. It redefines the loss function and Weak evaluator, and the integration method of the lifting algorithm has been improved to achieve a balance between computing speed and model effect.

XGBoost

Objective function of XGBoost

The objective function in the gradient boosting algorithm is optional, and different objective functions can be selected for classification or regression problems. As long as it is differentiable and can represent some kind of loss.
In addition to the traditional loss function, XGB also introduces model complexity to measure the computational efficiency of the algorithm. Therefore the objective function is written: traditional loss function + model complexity.
$$Obj=\sum _{i=1}^{m}l\left(y_i,{\hat{y}}_i\right)+\sum _{k=1}^{K}Oh\left(f_k\right)$$
part$i$ represents the iithitem in the data set$i$ samples,$m$ means import$The\; total\; amount\; of\; data\; for\; k$ trees, K represents the sum of all trees established so far (n_estimators), the first term is the traditional loss function that measures the difference between the true label and the predicted value, and the second term represents the complexity of the model. Using some transformation of the tree model$\Omega$ means. When we iterate each tree, we minimize$Obj$ to strive to obtain the optimal$\hat{y}$, thus minimizing the error rate of the model and the complexity of the model at the same time.
The loss function can also be understood from another perspective: through variance and bias. Generally speaking, variance and bias go hand in hand. The greater the variance, the smaller the bias.

Variance can be explained simply as the stability of the model across different data sets, while bias is the accuracy of the model's predictions. The variance-deviation dilemma can correspond to our $Objone$ O bj = ∑ i = 1 ml ( yi , y ^ i ) + ∑ k = 1 K Ω
$$Obj=\sum _{i=1}^{m}l\left(y_i,{\hat{y}}_i\right)+\sum _{k=1}^{K}Oh\left(f_k\right)$$
The first term is a measure of our bias, and the more inaccurate the model, the larger the first term will be. The second item is to measure our variance. The more complex the model, the more specific the model's learning will be, and the performance on different data sets will vary greatly, and the greater the variance will be. Therefore, the minimum value we solve is actually to find the balance point between variance and bias, so as to achieve the smallest generalization error of the model and the fastest running speed. We know that tree models and tree ensemble models are very easy to overfit, so most tree models will initially appear in the upper right part of the image, and we must control the model not to overfit through pruning. Now XGBoost's loss function has a part that limits the variance from becoming larger, which means that XGBoost will be smarter than other tree models and will not easily fall to the upper right of the image.

Solve the objective function of XGB

In the past, when solving the objective function, gradient descent or dual construction methods were usually used. However, there are no parameters that need to be solved in the objective function of XGB, so traditional gradient descent cannot be used. In XGB, tree $f_k$is not a vector of numbers and is the same as the input training feature matrix $x$ is not directly related, and although the iterative process can be compared to gradient descent, the solution process is completely different.
In the process of solving XGB, we strive to transform the objective function into a simpler writing method that is directly related to the structure of the tree, so as to establish the relationship between the structure of the tree and the effect of the model (including generalization ability and running speed). contact directly. Because of the existence of this connection, the objective function of XGB is also called the "structure score".
Our objective function is:
$$Obj=\sum _{i=1}^{m}l\left(y_i,{\hat{y}}_i\right)+\sum _{k=1}^{K}Oh\left(f_k\right)$$
at$t$次迭代得到的预测结果为：
$$\begin{array}{rl}{\hat{y}}_i^{(t)}=\sum _k^t{f}_k\left(x_i\right)& =\sum _k^{t-1}{f}_k\left(x_i\right)+f_t\left(x_i\right)\\ & ={\hat{y}}_i^{(t-1)}+f_t\left(x_i\right)\end{array}$$
So the first term of the objective function can be written as:
$$\begin{array}{rl}& \sum _{i=1}^{m}l(y_i,\widehat{y_i})\\ & =\sum _{i=1}^{m}l(y_i^t,{\widehat{y_i}}^{t-1}+f_t(x_i))\end{array}$$
And because for a certain batch of data, $y_i^t$is a known constant, so $The\; l$ function is equivalent to the following$F$函数：
$$F({\hat{y}}_i^{t-1}+f_t(x_i\left)\right)$$
Perform Taylor expansion of the F function. Note that although F here seems to be a unary function, it is actually related to the input data. However, for a certain batch of data, it is unary, but it is actually related to the input data. , so the l function is essentially a binary function. Therefore, partial derivation is used here instead of total derivation. There are many details to understand here, so I won’t go into details.
$$\begin{array}{rl}F\left({\hat{y}}_i^{(t-1)}+f_t\left(x_i\right)\right)& \approx F\left({\hat{y}}_i^{(t-1)}\right)+f_t\left(x_i\right)\ast \frac{\partial F\left({\hat{y}}_i^{(t-1)}\right)}{\partial {\hat{y}}_i^{(t-1)}}+\frac{1}{2}{\left(f_t\left(x_i\right)\right)}^2\ast \frac{{\partial }^{2F}\_\left({\hat{y}}_i^{(t-1)}\right)}{\partial {\left({\hat{y}}_i^{(t-1)}\right)}^2}\\ & \approx l\left(y_i^t,{\hat{y}}_i^{(t-1)}\right)+f_t\left(x_i\right)\ast \frac{\partial l\left(y_i^t,{\hat{y}}_i^{(t-1)}\right)}{\partial {\hat{y}}_i^{(t-1)}}+\frac{1}{2}{\left(f_t\left(x_i\right)\right)}^2\ast \frac{{\partial }^{2}l\left(y_i^t,{\hat{y}}_i^{(t-1)}\right)}{\partial {\left({\hat{y}}_i^{(t-1)}\right)}^2}\\ & \approx l\left(y_i^t,{\hat{y}}_i^{(t-1)}\right)+f_t\left(x_i\right)\ast g_i+\frac{1}{2}{\left(f_t\left(x_i\right)\right)}^2\ast h_i\end{array}$$
Now the first term of the loss function can be simplified to the following form:
$$\sum _{i=1}^m\left[l\left(y_i^t,{\hat{y}}_i^{(t-1)}\right)+f_t\left(x_i\right)g_i+\frac{1}{2}{\left(f_t\left(x_i\right)\right)}^2{h}_i\right)]$$
Now let’s discuss the current form of the loss function. We are now in the t-th iteration, so for the t-th iteration, the results obtained in the first t-1 iterations are constants, so$l(y_i^t,{\hat{y}}_i^{(t-1)})$ is a constant,$g_i$和$h_i$is also a constant, because our Taylor expansion is at ${\hat{y}}_i^{(t-1)}$expanded everywhere. $g_i$和$h_i$Also known as the per-sample gradient statistic, each sample is different.

Let’s look at the second term of the objective function: , at the t-th iteration
$$\sum _{k=1}^{K}Oh(f_k)=\sum _{k=1}^{t-1}Oh\left(f_k\right)+Oh\left(f_t\right)$$
, so the sum of the first term is equivalent to a constant, and only the latter term is related to the current iteration.
It should be mentioned here that when performing Taylor expansion, we made an assumption, that is,$f_t(x_i)$ is a small amount, and for tree models, there are generally hundreds of n-estimators, so this assumption is established, so the final form of our loss function is as follows:
$$Obj=\sum _{i=1}^m\left[f_t\left(x_i\right)g_i+\frac{1}{2}{\left(f_t\left(x_i\right)\right)}^2{h}_i\right)]+Oh\left(f_t\right)$$
The core part of this formula is that we need to determine$f_t$specific form.

Parameterized decision tree $f_t(x)$

For XGB, there is a prediction score on each leaf node, also called leaf weight. All samples falling on this leaf node have the same value, use $f_k\left(x_i\right)$ or$w$ means.
When there are multiple trees, the regression result of the ensemble model is the sum of the prediction scores of all trees. Assuming that there are a total of K decision trees in this ensemble model, the entire model will be in sampleii.The prediction result on$i$
$$\widehat{y_i}=\sum _{k=1}^K{f}_l\left(x_i\right)$$
Based on this understanding, let's consider each tree. For each tree, it has its own unique structure. This structure refers to the number of leaf nodes, the depth of the tree, the position of the leaves, etc., forming a tree structure that can define a unique model. In this structure, we use$q(x_i)$ represents sample$x_i$The leaf node where it is located, and use $w_{q(x_i)}$to express the $The\; i$ sample falls to thettthThe qth (xi) q(x_i)on$the\; t\; tree$$q\left(x_i\right)$ leaf nodes, then there is:
$$f_t\left(x_i\right)=w_{q\left(x_i\right)}$$
Suppose a tree contains a total of $T$ leaf nodes, where the index of each node is$j$ , then the sample weight of this leaf node is$w_j$. Based on this, define the complexity of the model $\Omega \left(f\right)$ is:
$$Oh\left(f\right)=\gamma T+Regularterm\; L2regularization$$
can be used:
$$\begin{array}{l}=\gamma T+\frac{1}{2}\lambda \Vert w{\Vert }^2\\ =\gamma T+\frac{1}{2}l{\sum }_{j=1}^T{w}_j^2\end{array}$$
使用L1正则：
$$\begin{array}{l}=\gamma T+\frac{1}{2}\alpha |w|\\ =\gamma T+\frac{1}{2}a{\sum }_{j=1}^T\left|w_j\right|\end{array}$$
Can also be used together:
$$=\gamma T+\frac{1}{2}a\sum _{j=1}^T|w_j|+\frac{1}{2}l\sum _{j=1}^T{w}_j^2$$
There are two parts in this structure. One part is $\gamma T$ , the other part is our regular term. Number of$T$ can represent the entire tree structure. This is because all trees in XGBoost are CART trees (binary trees), so we can$T$ determines the depth of the tree, and$\gamma$ is our self-defined parameter to control the number of leaves.
The parameters in front of the regularization term are used to control the regularization intensity. The same is true in XGB. When$\lambda$ and$The\; larger\; \alpha$ , the heavier the penalty, and the greater the proportion of regular terms. In the optimization direction of minimizing the objective function, the number of leaf nodes will be suppressed, and the complexity of the model will become lower and lower, so For XGB, which is inherently overfitting, regularization can improve the model effect to a certain extent.
In practical applications, regularization parameters are often not the best choice for us to adjust parameters. If we really want to control the complexity of the model, we will adjust$\gamma$ instead of adjusting these two regularization parameters. For tree models, pruning parameters still have a higher status.

Find the best tree structure: $w$ and$T$

We now express the tree in terms of prediction scores at leaf nodes, and the complexity of the tree is the number of leaves plus the regularization term:
$$f_t\left(x_i\right)=w_{q\left(x_i\right)},Oh\left(f_t\right)=\gamma T+\frac{1}{2}l\sum _{j=1}^T{w}_j^2$$
Suppose our $The\; structure\; of\; the\; t$ -tree has been determined as$q$ , we can bring the structure of the tree into our loss function to continue transforming the objective function. The purpose of transformation is to establish the relationship between the tree structure (number of leaf nodes) and the objective function. XGB uses L2 regularization by default.
$$\begin{array}{rl}& \sum _{i=1}^m\left[f_t\left(x_i\right)g_i+\frac{1}{2}{\left(f_t\left(x_i\right)\right)}^2{h}_i\right)]+Oh\left(f_t\right)\\ =& \sum _{i=1}^m\left[w_{q\left(x_i\right)}{g}_i+\frac{1}{2}{w}_{q\left(x_i\right)}^2{h}_i\right]+\gamma T+\frac{1}{2}l\sum _{j=1}^T{w}_j^2\\ =& \sum _{i=1}^m{w}_{q\left(x_i\right)}{g}_i+\left[\sum _{i=1}^m\frac{1}{2}{w}_{q\left(x_i\right)}^2{h}_i\right]+\gamma T+\frac{1}{2}l\sum _{j=1}^T{w}_j^2\end{array}$$
Now we still calculate the loss function by summing the number of samples. The current transformation goal is to calculate the loss function by summing the total number of leaf nodes.
Suppose we now have 2 leaves and 3 samples:

then the first term of the loss function can be calculated in the following way:
$$\begin{array}{rl}\sum _{i=1}^m{w}_{q\left(x_i\right)}\ast g_i& =w_{q\left(x_1\right)}\ast g_1+w_{q\left(x_2\right)}\ast g_2+w_{q\left(x_3\right)}\ast g_3\\ & =w_1\left(g_1+g_2\right)+w_2\ast g_3\\ & =\sum _{j=1}^T\left(w_j\sum _{i\in I_j}{g}_i\right)\end{array}$$
What is used here is wj w_j on each leaf node$w_j$( $j$ is the index subscript of the leaf node, we define the index asjjThe set of all samples on the leaf of$j$$I_j$则损失函数可以进一步被转化为：
$$\begin{array}{rl}Obj& =l\sum _{j=1}^T\left(w_j\ast \sum _{i\in I_j}{g}_i\right)+\frac{1}{2}\sum _{j=1}^T\left(w_j^2\ast \sum _{i\in I_j}{h}_i\right)+\gamma T+\frac{1}{2}l\sum _{j=1}^T{w}_j^2\\ & =\sum _{j=1}^T\left[w_j\sum _{i\in I_j}{g}_i+\frac{1}{2}{w}_j^2\left(\sum _{i\in I_j}{h}_i+l\right)\right]+\gamma T\end{array}$$
我们再定义：
$$G_j=\sum _{i\in I_j}{g}_i,H_j=\sum _{i\in I_j}{h}_i$$
So the loss function can be transformed into the following form:
$$\begin{array}{r}Ob{j}^{(t)}=\sum _{j=1}^T\left[w_j{G}_j+\frac{1}{2}{w}_j^2\left(H_j+l\right)\right]+\gamma T\\ F^{\ast }\left(w_j\right)=w_j{G}_j+\frac{1}{2}{w}_j^2(H_j+l\end{array}$$
where each $The\; value\; of\; j$ is always one with$w_j$is the quadratic function F ∗ F^* of the independent variable$F^{\ast }$ , our goal is to pursue$Obj$ is the smallest, as long as each leaf$The\; quadratic\; functions\; under\; the\; value\; of\; j$ are all smallest, so their sum must also be the smallest. So, we are at$F\ast$ on$w_j$Derivative, let the first derivative equal to 0 to find the extreme value, we can get:
$$\begin{array}{rl}\frac{\partial F^{\ast }\left(w_j\right)}{\partial w_j}& =G_j+w_j\left(H_j+l\right)\\ 0& =G_j+w_j\left(H_j+l\right)\\ w_j& =-\frac{G_j}{H_j+l}\end{array}$$
We find the number of leaf nodes TT in a givenThe minimum value of the loss function in the case of$T$$T$ is also a variable to be solved. By bringing this result into the objective function, we can get:
$$\begin{array}{rl}Ob{j}^{(t)}& =\sum _{j=1}^T[-\frac{G_j}{H_j+l}\ast G_j+\frac{1}{2}{\left(-\frac{G_j}{H_j+l}\right)}^2\left(H_j+l\right)\\ & =\sum _{j=1}^T\left[-\frac{G_j^2}{H_j+l}+\frac{1}{2}\ast \frac{G_j^2}{H_j+l}\right]+\gamma T\\ & =-\frac{1}{2}\sum _{j=1}^T\frac{G_j^2}{H_j+l}+\gamma T\end{array}$$
At this point, our objective function has changed dramatically. Our sample $i$ has been attributed to each leaf, and our objective function is calculated based on each leaf node, that is, the structure of the tree. Therefore, our objective function is also called "structure score". The lower the score, the better the overall structure of the tree. In this way, we have established a direct connection between the structure (leaves) of the tree and the effect of the model.
Let’s look at a specific example:

The calculation formula for the objective function of this tree is:
$$Obj=-\left(\frac{g_1^2}{h_1+l}+\frac{g_4^2}{h_4+l}+\frac{{\left(g_2+g_3+g_5\right)}^2}{h_2+h_3+h_5+l}\right)+3\gamma We$$
will find the best tree structure based on the loss function. It can be seen from the formula that$\lambda$和$\gamma$ are all set hyperparameters,$G_j$and $H_j$They are all determined by the loss function and the prediction result under a specific structure ${\hat{y}}_i^{t-1}$decided jointly. So we minimize the objective function and what we solve is the number of leaves, so the essence is to solve the structure of the tree. That is, for each iteration, the best parameter TT
is first solved$T$ , and then solve$G_j$and $H_j$, and then the weight wj w_j on each leaf can be solved$w_j$, thus finding our optimal tree structure and completing an iteration.
Next we will consider how to find the optimal tree structure:

Finding the best branching method: Gain

The greedy algorithm refers to the algorithm that controls the local optimum to achieve the global optimum. The decision tree algorithm itself is a method using the greedy algorithm. As an integrated model of trees, XGB naturally thinks of using this method to calculate, so we believe that if each leaf is optimal, then the overall generated tree structure is optimal, so that we can avoid enumerating all possibilities tree structure.

How we calculate in decision trees: We use the Gini coefficient or information entropy to measure the impurity of leaf nodes after branching. The difference between the information entropy before branching and the information entropy after divide and conquer is called information gain. The one with the largest information gain The branches on the feature are selected by us, and when the information gain is lower than a certain threshold, the tree stops growing. In XGB, the method we use is similar: we first use the objective function to measure the quality of the tree's structure, and then let the tree grow from depth 0. Each time a branch is made, we calculate how much the objective function is reduced ( So the value of Gain is the original score minus the score after branching) . When the reduction of the objective function is lower than a certain threshold we set, the tree will stop growing.
For example:

It’s still the same example as before. For the leaf node of the middle node (is male?), our $T=1$ , the structure score of this node is: I = { 1 , 4 } G = g 1 + g 4 H = h 1 + h 4 S coremiddle = − 1 2 G 2 H + λ + γ \begin{aligned} I & = \{ 1,4 \} \\ G & = g _ { 1 } + g _ { 4 } \\ H & = h _ { 1 } + h _ { 4 } \\ Score_{middle}&= - \ frac { 1 } { 2 } \frac { G ^ { 2 } } { H + \lambda } + \gamma \end{aligned}IGHScoremiddle​​={ 1,4}=g1​+g4​=h1​+h4​=−21​H+lG2​+c
For the younger brother and sister nodes, there are:
$$\begin{array}{rl}Scor{e}_{sister}& =-\frac{1}{2}\frac{g_4^2}{h_4+l}+c\\ Scor{e}_{brother}& =-\frac{1}{2}\frac{g_1^2}{h_1+l}+\end{array}$$
Relatively smooth equivalents:
$$\begin{array}{rl}-Gain& =Scor{e}_{sister}+Scor{e}_{brother}-Scor{e}_{middle}\\ & =-\frac{1}{2}\frac{g_4^2}{h_4+l}+c-\frac{1}{2}\frac{g_1^2}{h_1+l}+c-\left(-\frac{1}{2}\frac{G^2}{H+l}+c\right)\\ & =-\frac{1}{2}\frac{g_4^2}{h_4+l}+c-\frac{1}{2}\frac{g_1^2}{h_1+l}+c+\frac{1}{2}\frac{G^2}{H+l}-c\\ & =-\frac{1}{2}\left[\frac{g_4^2}{h_4+l}+\frac{g_1^2}{h_1+l}-\frac{G^2}{H+l}\right]+c\\ & =-\frac{1}{2}\left[\frac{g_4^2}{h_4+l}+\frac{g_1^2}{h_1+l}-\frac{{\left(g_1+g_4\right)}^2}{\left(h_1+h_4\right)+l}\right]+\end{array}$$
After removing the negative sign, the expression of Gain can be obtained:
$$Gain=\frac{1}{2}\left[\frac{G_L^2}{H_L+l}+\frac{G_R^2}{H_R+l}-\frac{{\left(G_L+G_R\right)}^2}{H_L+H_R+l}\right]-\gamma$$
CART trees are all binary trees, so this formula can be generalized. Among them$G_L$and $H_L$Calculated from the left node (younger brother node), $G_R$and $H_R$Calculated from the node (sister node), and $(G_L+G_R)$ and$(H_L+H_R)$ is calculated from the intermediate nodes. For any branch, we can calculate it in this way. In reality, we will perform the above calculation on all branch points of all features, and then select the node that makes the objective function decrease the fastest for branching. We perform such calculations for each layer of each tree. Compared with the original gradient descent, practice has proven that this method of solving the optimal tree structure is faster and can perform well on large data.

Important parameters $c$

$\gamma$ is a penalty term that will be subtracted every time we add a leaf. The more we add, the more severely Gain will be punished. It can be used to prevent overfitting.
In XGB, we stipulate that as long as the difference between structural scores$Gain$ is greater than 0, that is, as long as the objective function can continue to decrease, we allow the tree to continue branching. In other words, our requirement for the reduction of the objective function is:
$$\begin{array}{r}\frac{1}{2}\left[\frac{G_L^2}{H_L+l}+\frac{G_R^2}{H_R+l}-\frac{{\left(G_L+G_R\right)}^2}{H_L+H_R+l}\right]-c>0\\ \frac{1}{2}\left[\frac{G_L^2}{H_L+l}+\frac{G_R^2}{H_R+l}-\frac{{\left(G_L+G_R\right)}^2}{H_L+H_R+l}\right]>\end{array}$$
In this way, we can directly set $The\; size\; of\; \gamma$ to stop the trees in XGB from growing. $\gamma$ is therefore defined as the minimum reduction in the objective function required for further branching at the leaf nodes of the tree.