Before introducing two algorithms, first introduce on Boosting two core issues :
1) How to change the weight or the probability distribution of the training data in each round?
By increasing those weights wrong sample points preceding a weak classifier, a division of the former reducing the weight of the sample, so that the classifier to have better effect on the misclassified data. Such sub-sample was wrong stand out, resulting in a new sample distribution.
2) to a combination of weak classifiers by what means?
By additive model the weak classifier linear combination , by way of example, AdaBoost weighted majority vote, i.e., a small weight increase error rate of the classifier, while reducing the weight of a large error rate classifier.
By lifting the tree fit residuals way gradually reduce the residual, the each step of the generated model superimposed to give the final model.
Here are the steps to achieve the algorithm:
AdaBoost algorithm
Algorithm to achieve:
1, if it is Adaboost classification, function model using CART classification tree; if it is Adaboost regression function model using CART regression trees.
2, loss of function of "exponential loss function"
3, for Boosting two issues need to be addressed, AdaBoost algorithm uses the following strategies:
After selecting the training data instead of using the weighted training samples randomly selected, so that the focus is focused on training the training data sample points more difficult;
The weak classifiers together, using a weighted voting mechanism instead of the average voting mechanism. Let classification weak classifier has a good effect greater weight, and poor classification classifier has less weight.
Gradient Boosting Algorithm
Algorithm to achieve:
1, the function model CART regression tree model
2, it is generally loss function "log loss function" or "exponential loss function"
Gradient Boosting gradient algorithm that is lifting scheme,
3, the gradient descent optimization algorithm
4, for Boosting two issues need to be addressed, Gradient
Boosting algorithm uses the following strategies:
As the residual data in a training of the weak classifiers, the establishment of each new weak classifiers to such residuals are before weak classifiers to reduce the gradient direction.
The weak classifiers together, using a mechanism instead of the average cumulative voting mechanism.