Simple and crude understanding and implementation of machine learning ensemble learning (III): Boosting the principle of integration, implementation, api introduction, GBDT, XGBoost, Taylor expansion

Integrated learning

learning target

• Understand two core tasks of solving major integrated learning
• You know the principles of integrated bagging
• We know the process of establishing a random decision tree forest
• Why do I need to know random with replacement (Bootstrap) sampling
• Random Forest algorithm to achieve application RandomForestClassifie
• We know the principle of boosting integration
• Know the difference between bagging and boosting the
• Learn gbdt implementation process
  

5.3 Boosting

1.boosting integration principle

1.1 What is boosting

With the accumulation of learning from weak to strong

In short: each adding a new weak learners, overall capacity will be raised

Representatives algorithm: Adaboost, GBDT, XGBoost

1.2 implementation:

1. Training a learner

2. Adjust the data distribution

3. The second training learners

4. Adjust the distribution data again

5. Click learner training, adjustment data distribution

6. The whole process of realization

key point:

How to confirm the voting weights?

How to adjust the data distribution?

[Image dump the chain fails, the source station may have security chain mechanism, it is recommended to save the picture down uploaded directly (img-pmkuoxHP-1583250504350) (... / images / boosting9.png)]

AdaBoost construction process Summary

bagging integrated integrated with boosting differences:

A difference: the data front

Bagging: data is sampled training;

Boosting: the importance of adjusting the data according to the results of the previous round of learning.

Difference between the two: casting votes

Bagging: All learner affirmative vote;

Boosting: learning is weighted vote.

Three differences: learning sequence

Bagging learning parallel, each learner no dependencies;

Boosting learning is a serial, learning the order.

Distinguish four: the main role

Bagging mainly used to improve generalization performance (solved fit, it can be said to reduce variance)

Boosting主要用于提高训练精度 （解决欠拟合，也可以说降低偏差）

1.3 api介绍

• from sklearn.ensemble import AdaBoostClassifier
  • api链接:https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html#sklearn.ensemble.AdaBoostClassifier

2 GBDT(了解)

梯度提升决策树(GBDT Gradient Boosting Decision Tree) 是一种迭代的决策树算法，**该算法由多棵决策树组成，所有树的结论累加起来做最终答案。**它在被提出之初就被认为是泛化能力（generalization)较强的算法。近些年更因为被用于搜索排序的机器学习模型而引起大家关注。

GBDT = 梯度下降 + Boosting + 决策树

2.1 梯度的概念(复习)

2.2 GBDT执行流程

如果上式中的hi(x)=决策树模型,则上式就变为:

GBDT = 梯度下降 + Boosting + 决策树

2.3 案例

预测编号5的身高:

编号	年龄(岁)	体重(KG)	身高(M)
1	5	20	1.1
2	7	30	1.3
3	21	70	1.7
4	30	60	1.8
5	25	65	?

第一步:计算损失函数,并求出第一个预测值:

第二步:求解划分点

得出:年龄21为划分点的方差=0.01+0.0025=0.0125

第三步:通过调整后目标值,求解得出h1(x)

第四步:求解h2(x)

… …

得出结果:

编号5身高 = 1.475 + 0.03 + 0.275 = 1.78

2.4 GBDT主要执行思想

1.使用梯度下降法优化代价函数；

2.使用一层决策树作为弱学习器，负梯度作为目标值；

3.利用boosting思想进行集成。

3.XGBoost【了解】

XGBoost= 二阶泰勒展开+boosting+决策树+正则化

• Face questions: What XGBoost understand, please explain in detail how it works

Answer points: second-order Taylor expansion, boosting, decision trees, regularization

Boosting : XGBoost enhance the idea of multiple use Boosting weak learners iterative learning

Second-order Taylor expansion : each round of study, XGBoost loss of function of the second-order Taylor expansion, the use of first and second order gradient optimization.

Decision Tree : In each round learning, XGBoost decision tree algorithm to optimize learning as weak.

Regularization : XGBoost prevent overfitting, value added penalty term loss function, the restriction tree leaf node and the number of leaf nodes in the decision tree optimization process.

4 What is the Taylor expansion [expand]

Taylor expansion, the more accurate the results