Generally speaking, the relationship between the two variables is very subtle. It is not enough to describe it with simple straight line and curve parameter equations, so non-parametric regression is needed at this time. The difference between non-parametric and parametric methods is whether there are any restrictions on the prediction before the analysis. For example, it is believed that there is a linear relationship between the feature and the response variable, which can be fitted by a linear equation. We only need to ask for the coefficients of the equation to be the parameters Methods, such as the aforementioned linear regression, polynomial regression, etc., and if you analyze directly from the data, it is a non-parametric method. Precisely because there is no limit, the curve obtained by non-parametric method fitting can better describe the relationship between variables, no matter how complicated the curve relationship is, it can be fitted.
Loess (locally weighted regression) is a non-parametric method for local regression analysis. It mainly divides the sample into small areas, performs polynomial fitting on the samples in the interval, and repeats this process to get the results in different intervals. Weighted regression curves, and finally connect the centers of these regression curves to form a complete regression curve. The specific process is as follows:
· Determine the number and location
of fitting points · Determine the k closest points with the fitting point as the center
· Calculate the weight of these k points
by weighting function · Perform polynomial fitting (primary or quadratic) by weighted linear regression
· Repeat the above steps for all fitting points
Regarding the determination of the weight, it needs to be explained here. First, it needs to determine the distance from the point in the interval to the fitted point. This distance refers to the distance on the x-axis. We also need to find the largest distance in the interval, and then return to other distances. One treatment:
w i ( x 0 ) = W ( ∣ x 0 − x i ∣ Δ ( x 0 ) ) w_i(x_0) = W(\frac{|x_0 - x_i|}{\Delta (x_0)})wi(x0)=W(Δ(x0)∣x0−xi∣)
This weight is the smaller the distance from the fitting point, so we need to do a conversion, such as using tricube weight function:
W ( u ) = ( 1 − u 3 ) 3 W(u) = (1 - u^3)^3W(u)=(1−u3)3
The exponent can be quadratic (B function) or cubic (W function). The cubic power reduces the surrounding weights faster and has a better smoothing effect. It is suitable for most distributions, but increases the variance of the residuals. Generally, Say, the first iteration will be more practical W function, the second iteration will choose B function.
Regarding the fitting of weighted linear regression to the scattered points in the interval, the reason why we adopt weighted linear regression instead of ordinary linear regression is because we consider the fitting point, and the value of the point near it is on the fitting line The influence of should be greater, and the influence of a point farther away is smaller, so when we define the loss function, we should give priority to reducing the error between the nearby points and the fitted straight line. This is what we need to add to the ordinary least squares method The reason for the weight, in fact, this is the weighted least squares method:
J ( a , b ) = 1 N ∑ i = 1 N w i ( y i − a x i − b ) 2 J(a,b) = \frac{1}{N} \sum_{i=1}^N w_i(y_i -ax_i -b)^2J(a,b)=N1i=1∑Nwi(yi−axi−b)2
It can be seen that after adding the weight to the loss function, we will consider more weighted points when minimizing the loss function, hoping that they will be better, so that the fitting result will naturally be more biased towards the weighted points. In other words, the scattered points that are closer to the fitting point have a greater impact on the fitted straight line.
The above is the basic idea of loess.