xgboost等集成学习原理推导

机器学习的一般模型

o b j (θ) = L (θ) + Ω (θ)

$obj(\theta) = L(\theta)+\Omega(\theta)$
目标函数等于损失函数加正则项。在boost中，为弱学习器的级联。约定训练集为X，个数为N，维度为M，

x_{i}

$x_i$ 为第i个样本，

y_{i}

$y_i$ 为第i个样本的标签，

{\hat{y}}_{i}^{(0)} = 0 {\hat{y}}_{i}^{(1)} = {\hat{y}}_{i}^{(0)} + f_{1} (x_{i}) {\hat{y}}_{i}^{(2)} = {\hat{y}}_{i}^{(1)} + f_{2} (x_{i}) {\hat{y}}_{i}^{(3)} = {\hat{y}}_{i}^{(2)} + f_{3} (x_{i}) . . . . . . {\hat{y}}_{i}^{(t)} = {\hat{y}}_{i}^{(t - 1)} + f_{t} (x_{i})

$\hat{y}^{(0)}_i=0\\ \hat{y}^{(1)}_i=\hat{y}^{(0)}_i+ f_1(x_i)\\ \hat{y}^{(2)}_i=\hat{y}^{(1)}_i+ f_2(x_i)\\ \hat{y}^{(3)}_i=\hat{y}^{(2)}_i+ f_3(x_i)\\ ......\\ \hat{y}^{(t)}_i=\hat{y}^{(t-1)}_i+ f_t(x_i)\\$
其中

{\hat{y}}_{i}^{(t)}

$\hat{y}^{(t)}_i$ 为第i个数据的第t级输出，

f_{t} (x_{i})

$f_t(x_i)$ 为第i个样本被第t级学习器拟合的残差。

有：

o b j_{i}^{t} (f_{t}, x_{i}) = L [y_{i}, {\hat{y}}_{i}^{(t)}] + Ω (f_{t}) + c o n s t a n t

$obj^t_i(f_t,x_i)=L[y_i,\hat{y}_i^{(t)}]+\Omega(f_t)+constant\\$
将L在

{\hat{y}}_{i}^{(t - 1)}

$\hat{y}_i^{(t-1)}$ 处taylor展开，得：

L [y_{i}, {\hat{y}}_{i}^{(t)}] = L [y_{i}, {\hat{y}}_{i}^{(t - 1)}] + g_{i} f_{t} (x_{i}) + \frac{1}{2} h_{i} f_{i}^{2} (x_{i}) g_{i} = \frac{\partial L [y_{i}, {\hat{y}}_{i}^{(t - 1)}]}{\partial {\hat{y}}_{i}^{(t - 1)}} h_{i} = \frac{\partial^{2} L [y_{i}, {\hat{y}}_{i}^{(t - 1)}]}{\partial ({\hat{y}}_{i}^{(t - 1)})^{2}}

$L[y_i,\hat{y}_i^{(t)}]=L[y_i,\hat{y}_i^{(t-1)}]+g_if_t(x_i)+\dfrac{1}{2}h_if_i^2(x_i)\\ g_i=\dfrac{\partial L[y_i,\hat{y}_i^{(t-1)}]}{\partial \hat{y}_i^{(t-1)}}\\ h_i = \dfrac{\partial^2 L[y_i,\hat{y}_i^{(t-1)}]}{\partial (\hat{y}_i^{(t-1)})^2}\\$
定义

f_{t} (x_{i}) = w_{q} (x_{i})

$f_t(x_i)=w_q(x_i)$ ，其中

w_{q} (x_{i})

$w_q(x_i)$ 为

x_{i}

$x_i$ 落入的叶子节点的值（score），编号为q，值就是

w_{q}

$w_q$ 。对于落入相同叶子结点的，

w_{q} (x_{i})

$w_q(x_i)$ 的值相等。

对于所有的样本，有：

o b j^{t} (f_{t}) = \sum_{i = 0}^{N - 1} (L [y_{i}, {\hat{y}}_{i}^{(t)}]) + Ω (f_{t}) + c o n s t a n t Ω = γ T + λ \sum_{q = 0}^{T - 1} w_{q}^{2}

$obj^t(f_t)=\sum_{i=0}^{N-1}( L[y_i,\hat{y}_i^{(t)}])+\Omega(f_t)+constant\\ \Omega = \gamma T+\lambda \sum_{q=0}^{T-1}w^2_q\\$
则有：

L [y_{i}, {\hat{y}}_{i}^{(t)}] = L [y_{i}, {\hat{y}}_{i}^{(t - 1)}] + g_{i} f_{t} (x_{i}) + \frac{1}{2} h_{i} f_{i}^{2} (x_{i}) o b j^{t} (f_{t}) = \sum_{i = 0}^{N - 1} {g_{i} f_{t} (x_{i}) + \frac{1}{2} h_{i} f^{2} (x_{i})} + γ T + λ \frac{1}{2} \sum_{q = 0}^{T - 1} w_{q}^{2} + c o n s t a n t o b j^{t} (f_{t}) = \sum_{q = 0}^{T - 1} {\sum_{i \in I_{q}} g_{i} w_{q} + \frac{1}{2} \sum_{i \in I_{q}} h_{i} w_{q}^{2} + \frac{1}{2} λ w_{q}^{2}} + γ T + c o n s t a n t o b j^{t} (f_{t}) = \sum_{q = 0}^{T - 1} {G_{q} w_{q} + \frac{1}{2} (H_{q} + λ) w_{q}^{2}} + γ T

$L[y_i,\hat{y}_i^{(t)}]=L[y_i,\hat{y}_i^{(t-1)}]+g_if_t(x_i)+\dfrac{1}{2}h_if_i^2(x_i)\\ obj^t(f_t)=\sum_{i=0}^{N-1}\{ g_if_t(x_i)+\dfrac{1}{2}h_if^2(x_i)\}+\gamma T+\lambda\dfrac{1}{2}\sum_{q=0}^{T-1}w^2_q +constant\\ obj^t(f_t)=\sum_{q=0}^{T-1}\{\sum_{i\in I_q}g_iw_q+\dfrac{1}{2}\sum_{i\in I_q}h_iw_q^2+\dfrac{1}{2}\lambda w^2_q\}+\gamma T+constant\\ obj^t(f_t)=\sum_{q=0}^{T-1}\{G_qw_q+\dfrac{1}{2}(H_q+\lambda)w_q^2\}+\gamma T$
当树生成了之后，最优的

w_{q}^{*} = - \frac{G_{q}}{H_{q} + λ}

$w^*_q=-\dfrac{G_q}{H_q+\lambda}$ ,此时

o b j (f_{t}) = - \frac{G_{q}^{2}}{2 (H_{q} + λ)} + γ T

$obj(f_t)=-\dfrac{G_q^2}{2(H_q+\lambda)}+\gamma T$ 。对于不同的树有不同的损失，去最优，为了避免对所有的可能得树都进行生成后的比较，对分裂的叶子节点采用贪心算法:

G a i n = \frac{G_{L}^{2}}{H_{L} + λ} + \frac{G_{R}^{2}}{H_{R} + λ} - \frac{(G_{L} + G_{R})^{2}}{H_{L} + H_{R} + λ} - γ

$Gain =\dfrac{G_L^2}{H_L+\lambda}+\dfrac{G_R^2}{H_R+\lambda}-\dfrac{(G_L+G_R)^2}{H_L+H_R+\lambda}-\gamma$
遍历所有节点，选择增益最大值和属性作为分裂点。

即拟合残差时使用模型

{\hat{y}}_{i}^{(t)} = {\hat{y}}_{i}^{(t - 1)} + f_{t} (x_{i})

$\hat{y}^{(t)}_i=\hat{y}^{(t-1)}_i+ f_t(x_i)\\$
而加入使用的时候，在实际应用中，引入学习率，有模型：

{\hat{y}}_{i}^{(t)} = {\hat{y}}_{i}^{(t - 1)} + η f_{t} (x_{i})

$\hat{y}^{(t)}_i=\hat{y}^{(t-1)}_i+ \eta f_t(x_i)\\$
每次都不完全拟合残差，以防止过拟合。

例如，用 $L_i=\dfrac{1}{2}(y_i-\hat{y}_i)^2$ 作为损失函数，假设已经建立好了t-1颗树，则第t颗树

有：

g_{i} = \frac{\partial L [y_{i}, {\hat{y}}_{i}^{(t - 1)}]}{\partial {\hat{y}}_{i}^{(t - 1)}} = {\hat{y_{i}}}^{(t - 1)} - y_{i} h_{i} = \frac{\partial^{2} L [y_{i}, {\hat{y}}_{i}^{(t - 1)}]}{\partial ({\hat{y}}_{i}^{(t - 1)})^{2}} = 1

$g_i=\dfrac{\partial L[y_i,\hat{y}_i^{(t-1)}]}{\partial \hat{y}_i^{(t-1)}}=\hat{y_i}^{(t-1)}-y_i\\ h_i = \dfrac{\partial^2 L[y_i,\hat{y}_i^{(t-1)}]}{\partial (\hat{y}_i^{(t-1)})^2}=1\\$
但是前面的t-1颗树，都有

{\hat{y}}_{i}^{(t - 1)} = {\hat{y}}_{i}^{(t - 2)} + η f_{t - 1} (x_{i})

$\hat{y}^{(t-1)}_i=\hat{y}^{(t-2)}_i+ \eta f_{t-1}(x_i)$ ,由此计算

{\hat{y}}_{i}^{(t - 1)}

$\hat{y}_i^{(t-1)}$ 。

然后，通过最大增益生成树 $f_t$ 。

xgboost等集成学习原理推导

猜你喜欢