numpy-ml GBDT source code reading

CSDN

The formula for cross entropy is $-{\Sigma }_k^{K}p(y_k)\log p({\hat{Y}}_k)$，

对$\hat{Y}$After derivation, we get $-{\Sigma }_k^K\frac{p(y_k)}{p\left({\hat{Y}}_k\right)}$

Reflected in this formula:

numpy_ml.trees.losses.CrossEntropyLoss.grad

def grad(self, y, y_pred):
    eps = np.finfo(float).eps   # 对y_pred求导
    return -y * 1 / (y_pred + eps)

Note that for classification tasks, if there is $K$ classes, essentially training$K$ trees, then use OHE to classify$Y\in [0,K)\; is$ processed as k 0,1 column vectors. So for the kth component, the cross entropy degenerates to$-p(y)\log p(\hat{Y})$ 。

The negative gradient of the decision tree fitting at each step: $\frac{p(y)}{p(\hat{Y})}$

	$Y=0$	$Y=1$
$\hat{Y}=0$	0	$\infty$
$\hat{Y}=1$	0	1