3. Integration algorithm: RF, AdaBoost, GBDT, XGBoost, LightGBM, Stacking model fusion
1. Introduction to the principle of integration algorithm
The integration algorithm is to establish many weak evaluators (also called base evaluators), and then integrate the evaluation results of these weak evaluators according to certain integration rules, so as to achieve better results than a single weak evaluator. The core idea is that three cobblers are equal to Zhuge Liang.
Who is the Stooges? Decision trees, linear regression, logistic regression, naive Bayes, SVM, KNN, etc. are all available. Even these single evaluators can be used as base evaluators.
Can three cobblers stand up to one Zhuge Liang? If there are 3 weak evaluators, the effect of each weak evaluator is 0.6. Here we simply integrate based on the minority-majority integration rule. The accuracy after integration is 0.6*0.6*0.6+3*0.6*0.6 *0.4=0.6479999999999999. If 5 weak evaluators are integrated, the integrated accuracy is: 0.6*0.6*0.6*0.6*0.6+5*0.6*0.6*0.6*0.6*0.4+10*0.6*0.6*0.6*0.4*0.4=0.68256. If 7 are integrated. . . I don’t want to forget it, the accuracy after integration will definitely be higher.
Therefore, the integrated algorithm is still very powerful. Representative algorithms of integrated learning include: Random Forest, GBDT, Xgboost, LightGBM, etc. Just hearing the name sounds very powerful.
What integration rules are currently in place? There are three main integration methods:
Method 1: bagging method.
The typical representative of this method is random forest, which is also an introductory algorithm for ensemble algorithms. As the name suggests, the base evaluator of random forest is a decision tree, which means building multiple decision trees. If it is a classification task, it is integrated by the rule of the minority obeying the majority. If it is a regression task, it is integrated by the average method. But it’s not that simple, there are a few details here.
Detail 1. Each tree in the random forest is not built using all training samples. Instead, a part of the training samples are randomly selected and fit into a tree with replacement. If you think about it the other way around, if each tree uses all the samples, wouldn't each tree look the same? The prediction results of all trees will be the same, so wouldn't the final integrated result be the result of each tree? Therefore, when generating each tree, only some samples must be used to generate it.
Detail 2: In addition to sampling training samples, training features also need to be sampled! That is, the generation of each tree requires not only sampling samples but also sampling features. The purpose of this is to generate a variety of different trees. Each tree has different samples and characteristics, so each tree has its own different strengths. Therefore, when integrating using the bagging method, you must ensure that the base classifiers are independent of each other and different, so that the integration effect will be achieved.
Detail 3, this random sampling method with replacement, for small sample size data and data with few features, every sample and every feature is likely to be sampled and learned. But for a data set with a large training set sample size, some samples may not be sampled at all! If there are n samples, the probability of a sample being drawn is 1/n, and the probability of not being drawn is 1-1/n. If m times are drawn, the probability of not being drawn is (1-1 /n)**m, when m and n both tend to be very large, this probability tends to 1/e, that is, about 36.8% of the samples have not been drawn. These data are called out of bag data (oob), so when we use random forest, we do not need to divide the test set and training set, we only need to use out of bag data to test our model. Of course, the premise is that you have enough data and have out-of-bag data. When you don’t have a lot of data, there is no out-of-bag data, so it is naturally useless.
Summary: The bagging method can greatly reduce the risk of improper selection of a single model. Bagging can obtain a strong model with a smaller variance than its single base model. So when you use the bagging method, your base model is best to have a low bias and high variance base model. At this time, the integrated model will have smaller variance and the model will be more stable.
Method 2: boosting method.
The boosting method is the same as bagging. Its base models are all weak models of the same type. However, unlike bagging, each weak model is independent and connected in parallel. The base models in boosting are strictly ordered and connected in series.
Different from bagging, boosting focuses on generating a strong model with lower deviation than its single base model, which means screening and polishing each base model from front to back to make each base model very powerful, and also gives each base model Model different weights to integrate them. In layman's terms, it is to use various means to improve the effect of each model in the series, make each weak model very powerful, and integrate these base models in a weighted form, and naturally the final result will be even better. Based on this goal and direction, the boosting method has its own unique features in terms of sample selection, training process, and integration rules of the base model.
The boosting method is divided into two methods: adaptive boosting (adaboost) and gradient boosting (gradient boosting).
(1) adaboost is adaptive boosting. What is adaptive improvement? That is, in a series of base models (the order of the base models cannot be changed), the training set of each base model is the original training set, but the weight of each sample in the training set has changed, and this weight is based on it. The classification results of a classifier are automatically adjusted. It is to divide the samples classified by the previous classifier by a number, and multiply the incorrectly classified samples by this number. In this way, the sample weight of the data set changes, which is why it is adaptive. Then the training set with changed weights is fed into the next base model. We all know that the higher the weight of a sample, it means that the coefficient in front of its corresponding loss in the loss function is also greater. Under the guidance of the loss function, this sample will be focused on learning, so the previous classifier If the sample is wrong, the next classifier will focus on this sample. In addition to the different training sample weights of the base model, adaboost also uses weighting rules for integration. How to weight the integration? For example, the training results of the first base model are saved first. When starting to train the second base model, in addition to adjusting the sample weight for it, its training results are also saved, and then using linear regression, for example, to combine the two Put the prediction results and the real labels together and do a linear regression to get the integrated weight of the two base models. When starting to train the third base model, in addition to adjusting the sample weight, the results must be put together with the results of the first two base models and regressed against the real labels. If the third base model has no effect on fitting the real labels, If so, then its regression coefficient will naturally be 0, which means that the weight of the third base model is 0, and the integrated prediction result will have nothing to do with it. Continue to generate the fourth base model, and so on, ensuring that all The added base models all contribute to the fitting labels. This is also the reason for boost. A typical representative algorithm of adaptive boosting is AdaBoost. Since different tasks use different integration rules, AdaBoost has variants such as LogitBoost (classification) or L2Boost (regression).
(2) gradient boosting, gradient boosting, what is gradient boosting? Gradient boosting does not transform sample weights like adaboost, but the subsequent base model learns the residuals of the previous model. When the residuals reach a certain threshold, it stops generating new base models, so it is called gradient boosting. This method does not involve any integration rules. The predicted value of the integrated model is the sum of the predicted values of all base models. The effect of this method is very good. It is the same as the idea of residual network in deep learning, which is to directly fit the residual, and the effect is very good. The corresponding typical algorithms are GBDT and its improved and optimized engineering variant XGBoost, as well as the currently popular Microsoft open source LightGBM, which is also a gradient boosting framework based on the decision tree algorithm. LightGBM has quickly become the new favorite among current machine learning algorithms due to its amazing speed, distributed support, and small memory footprint.
Summary: When you use the boosting method, your base model must be some weak model, that is, a model that underfits your data. At this time, using boosting integration, the effect will be greatly improved. On the contrary, if your base model is an overfitting model, there is no need for boosting. There is no room for boosting. Secondly, the boosting method is sensitive to noise samples and abnormal samples. When the data set is not very good, your focus should be on feature engineering first. Do some tricks on feature engineering first, and then use the boosting method.
Method 3: stacking method.
Stacking uses different base evaluators, and then when integrating, follow the integration method in adaboost and perform linear regression or logistic regression on the results of all base evaluators.
Summary: The base evaluators in the bagging method are all the same, and their status is also the same. They are all independent, so they can be run in parallel. The base evaluators in the boosting method are also the same, but they are connected in series and cannot run in parallel. The base evaluator of stacking can also be trained in parallel, but it needs to be regressed in the end.
2. Prepare data: HiggsBoson data set
(1) Download data from kaggle
(2) Read the data, simply explore the data, and divide it into training sets and test sets
import time
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
data = pd.read_csv(r'training.csv')
features = data.iloc[:, 1:-2] #250000 rows × 31 columns
label = data.iloc[:, -1]
xtrain, xtest, ytrain, ytest = train_test_split(features, label, test_size=0.3, random_state=123)
3. Run all models first to see the bottom line of the data.
(1) Decision tree
start = time.time()
from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier()
tree.fit(xtrain, ytrain)
ypred = tree.predict(xtest)
score_train = tree.score(xtrain, ytrain)
score_test = tree.score(xtest, ytest)
print('训练集：', score_train, ' ', '测试集：', score_test)
print(accuracy_score(ypred, ytest))
print(time.time()-start)Training set: 1.0 Test set: 0.76444 0.76444 21.315803289413452
The decision tree is not an ensemble algorithm, but it is listed here because the base models of the subsequent ensemble models are all decision trees, so for comparison, let's look at the effect of a single tree model.
Judging from the score, a single tree predicts all predictions correctly for the training set, but its accuracy for the test is only 0.76444. It can be seen that this model is a typical overfitting model. Of course, the main reason is that we don’t adjust parameters at all, and all are default parameters. If we adjust the hyperparameters well and reduce overfitting, the model may be better, but the model limit is about the same.
(2) Random forest
start = time.time()
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(n_estimators=100)
clf.fit(xtrain, ytrain)
ypred = clf.predict(xtest)
score_train = clf.score(xtrain, ytrain)
score_test = clf.score(xtest, ytest)
print('训练集：', score_train, ' ', '测试集：', score_test)
print(accuracy_score(ypred, ytest))
print(time.time()-start)Training set: 0.9999885714285714 Test set: 0.8369866666666667 0.8369866666666667 215.39269065856934
Random forest directly increases the score from 0.76 to 0.83, but this model is also a typical overfitting model, and the training set is almost 100% fitted. But the main thing about this model is that it is too slow. It took more than 3 minutes.
（3）AdaBoost
start = time.time()
from sklearn.ensemble import AdaBoostClassifier
ada = AdaBoostClassifier(n_estimators=100)
ada.fit(xtrain, ytrain)
ypred = ada.predict(xtest)
score_train = ada.score(xtrain, ytrain)
score_test = ada.score(xtest, ytest)
print('训练集：', score_train, ' ', '测试集：', score_test)
print(accuracy_score(ypred, ytest))
print(time.time()-start)Training set: 0.8186114285714285 Test set: 0.8164266666666666 0.8164266666666666 158.35273146629333
Compared with random forest, adaboost is not good at fitting. The scores of the training set and the test set are almost the same. The test set is only slightly worse than the training set, which is very normal. Although the speed is improved compared to random forest, it is still too slow.
（4）GBDT
start = time.time()
from sklearn.ensemble import GradientBoostingClassifier
gbdt = GradientBoostingClassifier(n_estimators=100)
gbdt.fit(xtrain, ytrain)
ypred = gbdt.predict(xtest)
score_train = gbdt.score(xtrain, ytrain)
score_test = gbdt.score(xtest, ytest)
print('训练集：', score_train, ' ', '测试集：', score_test)
print(accuracy_score(ypred, ytest))
print(time.time()-start)Training set: 0.8342742857142857 Test set: 0.82964 0.82964 283.10672426223755
The effect of GBDT is a little better than that of ada, a little improved, and it is not overfitting, which is very normal. But the speed is too slow. There are many reasons for the slowness, but the effect is good, so the later xgboost and lightGBM were greatly optimized at the principle level and engineering level based on it.
(5) XGBoost
XGBoost can be said to be an engineered version of GBDT. You have to install it yourself: pip install xgboost==1.0.1 This version is recommended because it is more stable.
from xgboost import XGBClassifier
from sklearn.preprocessing import LabelBinarizer #处理标签列
#读取源数据
data = pd.read_csv(r'training.csv')
features = data.iloc[:, 1:-2] #250000 rows × 31 columns
label = data.iloc[:, -1]
xtrain, xtest, ytrain, ytest = train_test_split(features, label, test_size=0.3, random_state=123)
#给标签列编码
binarized = LabelBinarizer()
ytrain = binarized.fit_transform(ytrain).ravel()
ytest = binarized.fit_transform(ytest).ravel()
start = time.time()
#建模-训练-查看准确率
xgb = XGBClassifier(objective='binary:logistic')
xgb.fit(xtrain, ytrain)
ypred = xgb.predict(xtest)
score_train = xgb.score(xtrain, ytrain)
score_test = xgb.score(xtest, ytest)
print('训练集：', score_train, ' ', '测试集：', score_test)
print(accuracy_score(ypred, ytest))
print(time.time()-start)Training set: 0.8675142857142857 Test set: 0.8398933333333334 0.8398933333333334 3.4247934818267822
It can be seen that XGBoost is slightly better than GBDT, and it is not overfitting. The point is that the speed has been improved a lot, and the speed is almost one-tenth of GBDT.
(6) LightGBM
’s current new favorite LGB is not called an algorithm, but a framework. Since it is a framework, it means it is more complex and larger, so it needs to be downloaded and installed separately: pip install lightgbm.
The parameter description of light LGB can be written in a long blog. There are many online. Friends who want to dig out the details can Baidu other blog posts. , my focus here is to go through a complete process and understand the big framework, so I will not explain the details here. Of course, after you have a good understanding of the details of decision tree generation and pruning, as well as the operation ideas of boost integration, I believe that at the algorithm level, it should not be too difficult for you to understand LGB. Perhaps the biggest obstacle is the engineering optimization. A bit more obscure.
import time
import lightgbm
from sklearn.preprocessing import LabelBinarizer #处理标签列
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
#读取源数据
data = pd.read_csv(r'training.csv')
features = data.iloc[:, 1:-2] #250000 rows × 31 columns
label = data.iloc[:, -1]
xtrain, xtest, ytrain, ytest = train_test_split(features, label, test_size=0.3, random_state=123)
#给标签列编码
binarized = LabelBinarizer()
ytrain = binarized.fit_transform(ytrain).ravel()
ytest = binarized.fit_transform(ytest).ravel()
start = time.time()
# 将训练集和测试集整理成lgb要求的形式
lgb_train = lightgbm.Dataset(xtrain, ytrain) # 创建训练集，将数据保存到LightGBM二进制文件将使加载更快
lgb_test = lightgbm.Dataset(xtest, ytest, reference=lgb_train) # 创建测试集
# 将参数写成字典下形式
params = {'task': 'train', 'boosting_type': 'gbdt', 'objective': 'binary', 'metric':'auc', 'is_unbalance':True, 'force_col_wise':True}
# 训练模型
gbm = lightgbm.train(params,lgb_train,valid_sets=[lgb_train, lgb_test])
#查看训练结果
score = gbm.best_score
print('训练集：', score['training'])
print('测试集：', score['valid_1'])
# 训练后保存模型到文件
gbm.save_model('model_gbm.txt')
print(time.time()-start)[LightGBM] [Info] Number of positive: 59977, number of negative: 115023
[LightGBM] [Info] Total Bins 7388
[LightGBM] [Info] Number of data points in the train set: 175000, number of used features: 30
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.342726 -> initscore=-0.651171
[LightGBM] [Info] Start training from score -0.651171
训练集： OrderedDict([('auc', 0.9174424819661972)])
测试集： OrderedDict([('auc', 0.9077815342580077)])
5.596917152404785
It can be seen that a powerful algorithm is powerful. Only a few necessary and common parameters are set, and the details are not adjusted at all, and the score is directly raised to above 0.9! And the speed is also within an acceptable range.
(7) Stacking model fusion.
The previous integration algorithms, whether bagging or boosting, have the same basic models, that is, they are all tree models. But stacking is different. It fuses several heterogeneous models together, so the stacking method is a more powerful method and is a commonly used model fusion method in data competitions. Here I will simply run through a process to demonstrate.
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from sklearn.ensemble import StackingRegressor
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeRegressor
data = load_breast_cancer()
features = data.data
target = data.target
xtrain, xtest, ytrain, ytest=train_test_split(features,target,test_size=0.2)
model1 = DecisionTreeRegressor()
model2 = LinearRegression()
stacking = StackingRegressor(estimators=[('dt', model1), ('lr', model2)], final_estimator=LinearRegression())
stacking.fit(xtrain, ytrain)
# 预测
y_pred = stacking.predict(xtest)
# 模型评价
print(stacking)
rmse = mean_squared_error(ytest, y_pred) ** 0.5
rmseStackingRegressor(estimators=[('dt', DecisionTreeRegressor()),
('lr', LinearRegression())],
final_estimator=LinearRegression())
0.2441514442707548
4. Model tuning
Previously, the model only ran through and needed to be fine-tuned. Tuning is to combine a series of possible parameter values of the model to see which set of parameters is most suitable for our data, that is, which set of parameters can learn our data better, and which set of parameters can achieve higher scores on this data set. value, of course, this score must comprehensively consider the score of the training set and the score of the test set, that is, the over-fitting problem must be considered. Therefore, the main premise of model tuning is that, first, you must have a very clear understanding of the principles and mathematical calculations behind each model, so that you can have a direction for tuning. Second, you need to know some tuning tools. For example, at the end of my last decision tree article, I demonstrated two tuning methods, one is learning curve and the other is grid search. Obviously, for the 7 models that will be obtained in this article, the school curve will not work, because the learning curve is tuned for a single parameter, and it cannot be tuned in conjunction. The first six models in this article are recommended to use grid search. As for grid search tools, if your data is not huge and computer memory is sufficient, then you can use GridSearchCV in sklearn; if your data is There are a lot of them, you might as well try RandomizedSearchCV. If you think RandomizedSearchCV is not effective, then you try HalvingGridSearchCV.
Among the first 6 models in this article, except for the decision tree, the next 5 are integrated models. Although they are integrated models, they are all single models, but the seventh model is model fusion, and its tuning is more complicated. You can try HyperOPT optimization. You can continue with Baidu on how to use it, because it cannot be explained in a few sentences. Once you know this direction, you can explore it yourself.
At the end, I will give you a simple and popular introduction to the parameters of random forest. The premise of tuning is that you have a very good understanding of the parameters. Other models are also integrations of tree models. I believe you can guess by yourself. Eight, the only thing left is to read more data to confirm.
n_estimators 1. There must be several tree models in the forest, that is, several base classifiers.
The criterion is the calculation method of the impurity index. This index is the basis for branching of the tree model. A
splitter defaults to branching based on the calculation result of criterion. This parameter It can also be set to random division, in which case each tree will grow deeper. B
max_depth 2. The maximum depth of the tree C
min_samples_split 3. The minimum sample size for re-branching D
min_samples_leaf 4. The minimum sample size for leaf nodes E
min_weight_fraction_leaf The minimum weight sum of leaf nodes G
max_features 5. Several features should be selected to start with each branch Calculate and branch F
random_state random mode, similar to random number seed, you can enter any number. When we set this parameter, the same forest will grow after each instantiation for easy comparison. K
max_leaf_nodes The maximum number of leaf nodes, that is, it cannot be divided into too many leaf nodes. i
min_impurity_split The minimum impurity of the branch, that is, if the impurity is less than this parameter, there is no need to continue branching. j
class_weight The weight of the samples with various labels. When the samples are not This parameter should be considered during balancing, and the weight of a small number of labels should be increased to make the model more biased towards the minority class. H
The five parameters in the previous table 12345 are the five parameters with the highest frequency of adjustment. That is, we usually adjust these five parameters, and other parameters are adjusted according to the situation.
ABFK: criterion, splitter, max_features, random_state.
When using a tree to represent a two-dimensional table data (rows are samples, columns are features), a tree model is used to fit a two-dimensional table. When data is generated, these three parameters determine what kind of tree you will generate.
Criterion: When we generate a tree, the most important thing is to branch based on which value of that feature, that is, which feature is selected as the node, and which value of this feature is selected for branching. The basis is the impurity index, which is to calculate the impurity of each feature, and then branch the feature according to the lowest impurity index, and use this index to calculate which value of this feature is used to branch.
The parameter criterion is the calculation method of impurity. If it is a classification task, impurity can be calculated in two ways: information entropy or Gini Impurity (gini, also called information gain). If it is a regression task, impurity can be calculated using the mean square error mse or the mean absolute difference friedman_mse. So this parameter is the mathematical formula behind controlling how a tree grows.
Parameter splitter: There are two values. When splitter='best', after calculating the impurity of all features, find the indicator with the lowest impurity value and branch according to that indicator. But when we have tens of thousands of features, this method requires a lot of calculations, and the generation of the tree model is very slow. At this time, we can use this parameter splitter='random', which means randomly selecting a few features each time. Features are calculated and branched. But this situation is too random.
Therefore, when our splitter='best', we can also control the number of features calculated each time by adjusting the parameter max_features to reduce the amount of calculation.
The default max_features='None' means calculating all features; max_features='auto' or "sqrt" means only calculating the impurity of root N features each time; max_features='log2' means only calculating log2N features each time.
If your splitter=best, then you will grow the same tree every time. Because the data is unchanged, all features are used to calculate impurity, and all features are unchanged, so each time a branch is made based on which feature will remain unchanged, the resulting tree structure will also remain unchanged. But if your splitter='random' or max_features is not None, then the tree you generate will be different every time. Isn't it difficult to reproduce the model? In order to reproduce even if there is randomness, we use the random_state parameter. To control the random pattern every time it is randomized, so that the pattern is the same every time it is randomized, and it can be reproduced, which is convenient for you to compare parameters and adjust parameters.
C: max_depth: If ABFK can be combined to generate a complete tree, then max_depth is the parameter for violent pruning of the tree. This parameter limits the maximum depth of the tree. Limiting the depth can effectively suppress overfitting. For example, it is very extreme. The tree model I generated is subdivided into one sample and one result. The accuracy of the model in the training set must be 100%, but a tree with such deep and thin branches is an overfitting model. On the test set, It definitely won’t work, so we need to suppress the overfitting phenomenon of the model. Then this parameter is the most commonly used parameter.
ij: max_leaf_nodes, min_impurity_split: These two parameters also perform micro-violent pruning of the tree model from different angles. At present, it is naturally to suppress model overfitting. One is violent pruning from the number of leaf nodes; the other is pruning from the perspective of node impurity. When the impurity of the sample after a node is branched is less than the value of this parameter, do not branch anymore. Consider this node itself a leaf node. We generally do not use these two pruning methods. The default value of max_leaf_nodes is None, and the default value of min_impurity_split is 0. Just don't use these two parameters. The reason is that these two parameters are difficult to grasp when we are first modeling. Unless you have studied this data set for a long time and have built many tree models, you have a high understanding of the maximum number of leaf nodes and the minimum impurity. Once you have a grasp of it, you will know how to adjust these two parameters.
DE: min_samples_split, min_samples_leaf. If the max_depth parameter is violent pruning, and max_leaf_nodes and min_impurity_split are expert pruning, then these two parameters are the intensive pruning operations of us ordinary people.
For example, min_samples_split =5 means that if the number of samples in the current node is less than 5, even if the targets of these 5 samples are different, there is no need to continue branching. Just stop branching, that's all.
For example, min_samples_leaf=3 means that if the sample size in the current leaf node is less than 3, the leaf node and its sibling nodes will be deleted, leaving only the parent node.
D and E are often paired together. For example, if a node has 6 samples, then it meets the conditions for continuing the branch. However, if 2 of the 6 samples have a target of 1, and 4 of the samples have a target of 2, Then don't divide it, because the sample size of the left branch is 2, which is less than 3, so don't divide it. Let the nodes of these 6 samples be used as leaf nodes. Don't continue dividing.
These two parameters also prevent over-fitting. If they are set too large, it will prevent the model from continuing to learn. If the parameters are set too small, it will be over-fitting. These two parameters are used in conjunction with C, which has a good anti-overfitting effect.
GH: These two parameters, min_weight_fraction_leaf and class_weight, are used together, mainly for the imbalance of samples in classification problems. The default value of class_weight is None, which means that the sample balance problem is not considered, and the sample is basically balanced. When class_weight="balanced", the algorithm will automatically calculate the sample weights of various labels. At this time, various types of samples have different weight values. The default value of the parameter min_weight_fraction_leaf is 0, which means that the weight issue is not considered, that is, it does not matter what the weight value of the sample in the leaf node is. If min_weight_fraction_leaf =0.01 is set, it means the sum of the weights of all samples in the leaf node (the weight value of each sample is provided by class_weight). If it is less than 0.01, the leaf node will be pruned together with its sibling nodes.