learning background
Recently, I want to learn and implement the XGBoost algorithm, because I have some ideas about the project recently, and I am going to make a regression prediction. As the most popular regression prediction algorithm at the moment, we are going to apply the test effect directly.
1. Basic knowledge
(1) Taylor formula
Taylor's formula is a formula that uses the information of a function to describe the value near a point. Has local effectiveness.
The basic form is as follows:
From the above basic form, it can be known that the iterative form of Taylor's formula is:
The iterative form above is for second-order Taylor expansions, but you can also perform more-order Taylor expansions.
(2) Gradient descent method
In the machine learning algorithm, our goal is to minimize the loss function L(theta). Combined with the Taylor expansion formula, we can get the following formula:
Why here 
, I think it is because 
after it is multiplied with the previous one, as long as a takes a positive value, then its change value must be a negative number, in other words, it can ensure that his loss must be reduced.
(3) Newton's method
Newton's method, like gradient descent, is an optimization method. The difference between them is the second-order Taylor expansion used by Newton's method, and the first-order Taylor expansion used by gradient descent.
2. Concept Explanation
（1）gradient descend和gradient boosting
The difference between gradient descend and gradient boosting is that the former is the update of the parameters, and the latter is the update of the model
(2) Boosting algorithm
The Boosting algorithm is an additive model:
(3) Decision tree
Advantages of decision trees: interpretability, can handle mixed types of features, scalable and non-deformable, has the function of feature combination, can naturally deal with missing values, robust to outliers, has feature selection, strong scalability, easy parallel
Disadvantages of decision trees: lack of smoothness (for regression trees), not suitable for processing high-dimensional sparse data
3. GBDT algorithm
The XGBoost algorithm is an advanced version of the GBDT algorithm. Before we understand and implement the XGBoost algorithm, we must first understand the GBDT algorithm.