Machine learning notes (2) k nearest neighbor algorithm kNN, linear regression method Linear Regression

1. K nearest neighbor algorithm KNN

The k-neighbor algorithm, or the k-nearest neighbor (kNN, k-NearestNeighbor) classification algorithm is one of the simplest methods in data mining classification techniques. The so-called k nearest neighbors means the k nearest neighbors, which means that each sample can be represented by its closest k neighbors. The nearest neighbor algorithm is a method of classifying each record in the data set.

1.1. Core idea

The core idea of ​​the kNN algorithm is that if most of the K nearest neighbor samples of a sample in the feature space belong to a certain category, the sample also belongs to this category and has the characteristics of samples in this category. In determining the classification decision, this method only determines the category of the sample to be divided according to the category of the nearest one or several samples. The kNN method is only related to a very small number of adjacent samples when making category decisions. Since the kNN method mainly relies on the limited surrounding samples rather than the method of discriminating the class domain to determine the category to which it belongs, the KNN method is more accurate than other methods for the sample sets to be divided when the class domain crosses or overlaps more. for fit.

1.2. Algorithm process

In general, the KNN classification algorithm includes the following four steps:
① Prepare data and preprocess the data.
② Calculate the distance from the test sample point (that is, the point to be classified) to each other sample point.
③Sort each distance, and then select K points with the smallest distance.
④ Compare the categories of K points, and classify the test sample points into the category with the highest proportion among the K points according to the principle that the minority obeys the majority.

1.3. Definition of distance

• 欧拉距离
  $$\sqrt{\sum _{i=1}^n(X_i^{(a)}-X_i^{(b)}{)}^2}$$
• Manha顿Distance∑
  $$\sum _{i=1}^n|X_i^{\left(a\right)}-X_i^{\left(b\right)}|$$
• Minkowski distance
  $$(\sum _{i=1}^n|X_i^{\left(a\right)}-X_i^{\left(b\right)}{|}^p{)}^{\frac{1}{p}}$$
• Vector Space Cosine Similarity Cosine Similarity
• Adjusted Cosine Similarity Adjusted Cosine Similarity
• Pearson Correlation Coefficient
• Jaccard similarity coefficient Jaccard Coefficient

1.3. Advantages and disadvantages

Advantages
The KNN method is simple in thinking, easy to understand, easy to implement, and does not need to estimate parameters. The main
disadvantage
of this algorithm in classification is that when the samples are unbalanced, for example, the sample size of one class is large, while the sample size of other classes is small. , it may cause that when a new sample is input, among the K neighbors of the sample, the samples of the high-capacity class are in the majority.
Another disadvantage of this method is that the amount of calculation is large, because for each text to be classified, the distance to all known samples must be calculated to obtain its K nearest neighbors.

1.4. Parameters and model parameters

• Hyperparameters
  Parameters that need to be decided before the algorithm runs
• Model parameters
  Parameters learned during the algorithm process

The kNN algorithm has no model parameters and is a non-parametric learning algorithm. In the kNN algorithm, k is a typical parameter.

2. Linear Regression

2.1. Simple linear regression

Looking for a straight line, the relationship between the sample output marks that "fits" the sample characteristics to the greatest extent There

is only one sample characteristic, which is called: simple linear regression.

Suppose we find the best-fit equation of the line: $y=ax+b$ , then for each sample point$x^{\left(i\right)}$ , according to our line equation, the predicted value is:${\hat{y}}^{(i)}=a{x}^{\left(i\right)}+b$ , meaning:$y^{\left(i\right)}$ . We want$y^{(i)}$ 和 ${\hat{y}}^{\left(i\right)}$ The gap is as small as possible, which means$y^{\left(i\right)}$ 和 ${\hat{y}}^{\left(i\right)}$ distance:$y^{(i)}-{\hat{y}}^{\left(i\right)}$ , but this means that the gap is positive or negative, and it is obviously unreasonable to add all the sample gaps to 0, so that the gap is 0. To remove the negative sign, we can easily think of using$|y^{\left(i\right)}-{\hat{y}}^{\left(i\right)}|$means, but there is a problem with this way that x is not derivable everywhere. In order to be derivable everywhere and remove the negative sign, the gap can be used as the variance:$(y^{\left(i\right)}-{\hat{y}}^{(i)}{)}^2,$considering all samples:$\sum _{i=1}^m(y^{(i)}-{\hat{y}}^{(i)}{)}^2$ . Our goal is to make$\sum _{i=1}^m(y^{\left(i\right)}-{\hat{y}}^{\left(i\right)}{)}^2$ as small as possible, where${\hat{y}}^{\left(i\right)}=a{x}^{\left(i\right)}+b$ , then find a and b such that$\sum _{i=1}^m(y^{\left(i\right)}-a{x}^{\left(i\right)}-b{)}^2$ as small as possible.

2.1.1. The basic idea of ​​a class of machine learning algorithms

The goal is to find a and b such that $\sum _{i=1}^m(y^{(i)}-a{x}^{\left(i\right)}-b{)}^2$ as small as possible. By analyzing the problem, the loss function or utility function of the problem is determined. $\sum _{i=1}^m(y^{\left(i\right)}-a{x}^{\left(i\right)}-b{)}^2$ is called the loss function (loss function) or utility function (utility function). By optimizing the loss function or utility function, a machine learning model is obtained. Almost all parameter learning algorithms are like this (linear regression, polynomial regression, logistic regression, SVM, neural network...)
For simple linear regression, find a and b such that$\sum _{i=1}^m(y^{\left(i\right)}-a{x}^{\left(i\right)}-b{)}^2$ as small as possible is a typical least multiplication problem: minimize the variance of the error. $a=\frac{\sum _{i=1}^m(x^{\left(i\right)}-\bar{x})({and}^{\left(i\right)}-\bar{y})}{\sum _{i=1}^m(x^{(i)}-\bar{x}{)}^2}$  $b=\bar{y}-a\stackrel{ˉVectorized}{x}$
operation

$\sum _{i=1}^m(x^{(i)}-\bar{x})({and}^{\left(i\right)}-\bar{y})=\sum _{i=1}^m{w}^{\left(i\right)}\cdot v^{\left(i\right)}=w\cdot v$

$w=(w^{(1)},w^{(2)},\cdots ,w^{(m)})v=(v^{\left(1\right)},v^{\left(2\right)},\cdots ,v^{\left(m\right)})$

2.1.2. Measurement of regression algorithm

1. Mean Square Error MSE (Mean Squqraed Error)

$$\frac{1}{m}\sum _{i=1}^m(y_{test}^{(i)})-{\hat{y}}_{test}^{\left(i\right)}{)}^2$$

2. Root mean square error RMSE (Error of Root MeanSquare)

$$\sqrt{\frac{1}{m}\sum _{i=1}^m(y_{test}^{(i)})-{\hat{y}}_{test}^{\left(i\right)}{)}^2}=\sqrt{MS{E}_{test}}$$

3. Mean Absolute Error MAE (Mean Absolute Error)

$$\frac{1}{m}\sum _{i=1}^m|y_{test}^{\left(i\right)}-{\bar{y}}_{test}^{\left(i\right)}|$$

4. R Squareed

    $R^2=1-\frac{S{S}_{residual}}{S{S}_{total}}$

    $R^2=1-\frac{\sum _i(y^{(i)}-{\hat{y}}^{\left(i\right)}{)}^2}{\sum _i(\bar{y}-y^{(i)}{)}^2}=1-\frac{(\sum _i^m(y^{(i)}-{\hat{y}}^{\left(i\right)}{)}^2)/m}{(\sum _i^m(\bar{y}-y^{(i)}{)}^2)/m}=1-\frac{MSE(\hat{y},y)}{Var(y{)}_{(variance}}$

    $\sum _i(y^{(i)}-{\hat{y}}^{\left(i\right)}{)}^2$ is the error generated using our model prediction

    $\sum _i(y^{\left(i\right)}-{\hat{y}}^{\left(i\right)}{)}^2$ is to use$y=\bar{y}$(Baseline Model) prediction errors

• $R^2<=1$
• $R^2$ The bigger the better. When our predictive model makes no mistakes,$R^2$ gets the maximum value of 1
• When our model is equal to the baseline model, $R^2$ is 0
• If $R^2<0$ , indicating that the model we learned is not as good as the baseline model. at this time. Chances are that our data doesn't have any linear relationship.

2.2. Multiple linear regression

Sample features with multiple
targets: $\sum _{i=1}^m(y^{(i)}-{\hat{y}}^{(i)}{)}^2$ possible small, of which${\hat{y}}^{\left(i\right)}=i_0+i_1{X}_1^{(i)}+i_2{X}_2^{\left(i\right)}\cdots +i_n{X}_n^{(n)}$,inputsθ $i_0,i_1,i_2,\cdots ,i_n$，使得$\sum _{i=1}^m(y^{(i)}-{\hat{y}}^{(i)}{)}^2$ as small as possible. Convert it to vectorization:

${\hat{y}}^{\left(i\right)}=i_0{X}_0^{\left(i\right)}+i_1{X}_1^{\left(i\right)}+i_2{X}_2^{\left(i\right)}\cdots +i_n{X}_n^{\left(n\right)},X_0^{\left(i\right)}\equiv 1$

    $i=(i_0,i_1,i_2,\cdots ,i_n{)}^T$

    $X^{(i)}=(X_0^{(i)},X_1^{\left(i\right)},X_2^{\left(i\right)},\cdots ,X_n^{\left(i\right)})$

${\hat{y}}^{\left(i\right)}=X^{\left(i\right)}\cdot i$

$X_b=\left[\begin{array}{ccccc}1& X_1^{(1)}& X_2^{\left(1\right)}& \cdots & X_n^{\left(1\right)}\\ 1& X_1^{(2)}& X_2^{\left(2\right)}& \cdots & X_n^{\left(2\right)}\\ \cdots & & & & \cdots \\ 1& X_1^{(m)}& X_2^{\left(m\right)}& \cdots & X_n^{\left(m\right)}\end{array}\right]i=\left[\begin{array}{c}{i}_0\\ i_1\\ i_2\\ \cdots \\ i_n\end{array}\right]$

$\hat{y}=X_b\cdot i$

使$\sum _{i=1}^m(y^{(i)}-{\hat{y}}^{(i)}{)}^{2.}$ Let$(\hat{y}-X_b\cdot i{)}^T(\hat{y}-X_b\cdot \theta )$ , then you can find the normal equation solution (Normal Equation) of multiple linear regression
$$i=(X_b^T{X}_b{)}^{-1}{X}_b^{T}y$$
problem with this approach: high time complexity:$O\left(n^3\right)$(optimized O($n^{2.4}$ ))
advantage: no need to normalize the${}^{data}$

2.3. Polynomial regression

The multinomial regression analysis method that studies a dependent variable and one or more independent variables is called polynomial regression (Polynomial Regression). If there is only one independent variable, it is called univariate polynomial regression; if there are multiple independent variables, it is called multivariate polynomial regression. In univariate regression analysis, if the relationship between the dependent variable y and the independent variable x is nonlinear, but no appropriate function curve can be found to fit, then univariate polynomial regression can be used.
The unary m-degree polynomial regression equation is: $\hat{y}=b_0+b_{1}x+b_2{x}^2+\cdots +b_m{x}^m$ .
The binary quadratic polynomial regression equation is:$\hat{y}=b_0+b_1{x}_1+b_2{x}_2+b_3{x}_1^2+b_4{x}_2^2+b_5{x}_1{x}_2$.
The biggest advantage of polynomial regression is that the measured points can be approximated by increasing the high-order term of x until it is satisfied. In fact, polynomial regression can deal with quite a class of nonlinear problems, and it occupies an important position in regression analysis, because any function can be approximated by polynomials piecewise. Therefore, in common practical problems, regardless of the relationship between dependent variables and other independent variables, we can always use polynomial regression for analysis.