Learning Classification of Machine Learning

  This article is the third article of the "Mathematics of Machine Learning" column. After briefly introducing why there is machine learning and what machine learning can do, and learning regression, I will summarize the knowledge of learning classification in machine learning.

1. Set up the model

In order to facilitate the description of the problem, the size of the image is divided into two categories: horizontal and vertical, so this is a two-category problem.

1.1 Data

Here, 7 images are used as training data, as shown in the following table.

Width	high	shape
80	150	portrait
60	110	portrait
35	130	portrait
160	50	horizontal
160	20	horizontal
125	30	horizontal

2.1 Graphical description

Display the above 7 data in a two-dimensional coordinate system, as shown in the figure below.

Q1: How to classify?
A1: Our goal is to find a straight line that can divide the existing horizontal and vertical data into two parts. It is easy to know from the above figure that using a straight line can divide the two types of data into two parts. Suppose it is a straight line as shown in the figure below.

Q2: How to find this straight line?
A2: The method of finding a straight line in the regression part is to find the slope and intercept of the straight line (for details, please refer to the regression of machine learning ), but it is different in the classification part. We use it 向量and introduce it hereweight vector $w$, the goal is further specified, now the goal is to findLine with weight vector as normal vector。

Q3: What is a weight vector? A3: The weight vector is the parameter we require, and the parameter θ \theta
required for regression$same\; as\; \theta$ . Knowing the weight vector will know the straight line of classification, so we need to find the most suitable parameter vector$w$。

Q4: What is the mathematical relationship between the weight vector and the required classification line?
A4: $w\cdot x=0$ , that is, all vectors on the line are perpendicular to the weight vector. The weight vectors are shown in the figure below.

2. Inner product

Next, it mainly revolves around w ⋅ x = 0 w x = 0 with the background of machine learning$w\cdot x=The\; calculation\; of\; 0$ is expanded.

1. $w\cdot x={\sum }_{i=0}^n{w}_i\cdot x_i=0$ . In connection with the background of this question, since only the height and width of the image are considered, it istwo-dimensional problem, so here it can be expressed as $w\cdot x=w_1\cdot x_1+w_2\cdot x_2=0$。

   Suppose the w 1 = 1 w_1=1 calculated here$w_1=1$,$w_2=1$ , the expression is$x_1+x_2=0$ means$x_1=-x_2$. Therefore, as long as the weight vector is known, the straight line of classification is found. The obtained straight line and weight vector are graphically represented as follows:
   
   due to the relationship with wwThere are many vertical vectors of$w\; ,\; and\; they\; even\; form\; a\; straight\; line.$This is what it means to find the line that makes the weight vector a normal vector.

2. $w\cdot x=|w|\cdot |x|\cdot cos\theta =0$ , here due to$|w|$ and$|x|$ is a non-negative number, if result = 0, then$i=90^{.}$, or 9 0if the result is negative$90^{。}<i<270^{.}$, and 0 if the result is positive$0^{。}<i<90^{.}or270^{。}<i<360^{。}$

3. Perceptron

Q5: What is a perceptron?
A5: A perceptron is a model that receives multiple input variables and outputs the sum of the products of each variable and its respective weight. Its graphical representation is as follows:

   The above perceptron model is also calledsimple perceptronorsingle layer perceptron。

Q6: What are the advantages and disadvantages of the perceptron?
A6: The advantage is that the principle is simple, and it is the model basis for deep learning and neural networks. But the disadvantage is also obvious, that is, it can only be used to solve the problem of linear separability (see the fourth part for details). From the perspective of expression, it is $x$ can only be a one-time power, and most problems in reality are multidimensional and linearly inseparable, so they are rarely used in practical problems.

Q7: Why introduce the perceptron?
A7: Since the perceptron is to calculate the product of multiple variables and their weights and output the sum, it is consistent with the goal of the classification algorithm (to obtain the classification line from the weight vector).

hereUsing the Perceptron Model to Find the Target Straight Line. Before that, the training data needs to be prepared.

3.1 Prepare training data

   Classification is a method of using labeled data (labeled data can be understood as knowing the input and output of the model) to classify unknown data. Here the width of the image is set to $x_1$, the height is set to $x_2$, the label is set to 1 for landscape images and -1 for portraits. Then the training data is as follows:

image size	shape	$x_1$	$x_2$	$y$
80x150	portrait	80	150	-1
60x110	portrait	60	110	-1
35x130	portrait	35	130	-1
160x50	horizontal	160	50	1
160x20	horizontal	160	20	1
125x30	horizontal	125	30	1

   The next step is to 参数向量judge whether the image is landscape or portrait. A function that judges data like this is called 判别函数.
   Let's understand it from a graphical point of view first. In the figure below, on the upper left of the straight line is the vertical image, so the data in this range $(x_1,x_2)$ and the weight vector <90, that is,$w\cdot x^{(i)}\ge 0$ , as shown in the figure below.

  Similarly, the horizontal image is below the straight line, and the angle between it and the weight vector is greater than 90 and less than 270, that is,$w\cdot x^{\left(i\right)}<0$。可得到判别函数表达式如下：
$$f_w{x}^{(i)}=\{\begin{array}{rlrlr}1& & & & w\cdot x^{\left(i\right)}\ge 0\\ -1& & & & w\cdot x^{\left(i\right)}<0\end{array}$$

3.2 Update expression of weight vector

  With the discriminant function, we can update the parameter $w$ to find values ​​that classify the data well. Give the conclusion first. Here is the update expression for the weight vector:$$w:=\{\begin{array}{rlrlr}w+y^{(i)}\cdot x^{\left(i\right)}& & & & f_w(x^{(i)})\ne y^{(i)}\\ w& & & & f_w\left(x^{\left(i\right)}\right)=y^{(i)}\end{array}$$
   fw ( x ( i ) ) ≠ y ( i ) f_w(x^{(i)}) \ne y^ { $f_w\left(x^{\left(i\right)}\right)\ne y^{\left(i\right)}$ represents the result of the discriminant function$f_w\left(x^{\left(i\right)}\right)$and the real result$y^{\left(i\right)}$ dissimilarity, and$f_w\left(x^{\left(i\right)}\right)=y^{\left(i\right)}$ means that the results of the two are consistent. Observe this formula again, only when the results of the two are inconsistent$w$ is updated, when the two results are consistent$w$ remains unchanged.

Q8: Why when the results of the two are inconsistent, $w$ 's update formula is$w+y^{(i)}\cdot w\cdot x^{\left(i\right)}$ ?
A8: From a graphical point of view, the weight vector is randomly determined at the beginning (just like$The\; determination\; of\; \theta$ is also random), if it is the weight vector in the figure below.

   The target classification line can be obtained from the weight vector, as shown in the figure below.

   Now the data (80, 150) is judged by the discriminant function where the weight vector is located, and the data (80, 150) is in the position shown in the figure.

   The data is also represented as a vector, as shown in the figure below.

   For (80, 150) data:

1. The result of discriminant function $f_w{x}^i=w\cdot x^i$ , since the angle between the two is less than 90 degrees, so$f_w{x}^i=1$。
2. Since it is a vertical image, the real value $y^i=-1$
   From the above results, it can be seen that the classification result is different from the real value, so the weight vector needs to be updated. When the results of the two are different,wwThe update expression of$w$$w=w+y^i\cdot x^i$ . yi$y^i=-1$，cause$w=w-x^i$ . That is, the subtraction of vectors can be expressed in the figure as:
   
      the parallelogram rule is used, and the red vector in the figure is the updated weight vector.
      For (80, 150) data:
3. The result of discriminant function $f_w{x}^i=w\cdot x^i$ , since the angle between the two is greater than 90 degrees, so$f_w{x}^i=-1$。
4. Since it is a vertical image, the real value $y^i=-1$
   From the above results, it can be seen that the updated classification result is the same as the real value, and there is no need to update the weight vector, and then train the next data.A method like this that repeatedly updates all parameters is the perceptron model.

4. Linearly separable

   Linearly separable literally means classifying data using a straight line. The perceptron model can only solve linearly separable problems. This feature also limits the scope of use of the perceptron model, because real problems are mostly affected by multiple variables. Then it leads to a question, how to solve the linear inseparable problem?

5. Logistic regression

  When calculating the regression, we define the objective function, differentiate the objective function, obtain the update function of the parameters, use the data to update the parameters, and then use the obtained parameters to calculate the data $x$ for regression. This method is logistic regression. Logistic regression can also be used for classification, which is a method different from perceptrons, which divides the classification results fromprobabilityperspective. For the convenience of understanding, for the time being, only the height and width of the image are considered to divide the image into vertical and horizontal categories.

5.1 sigmoid function

  First introduce the sigmoid function, its form is as follows: $$f_{}(x)=\frac{1}{1+exp(-i^T\cdot x)}$$  where, $exp(-i^T\cdot x)$ is equivalent to$e^{i^T\cdot x}$, since$i^T\cdot x$ as an exponent may be unclear, so the former is used. The image of the sigmoid function is shown in the image:
  From the figure, two main characteristics of the sigmoid function can be observed:
    1.$i^T\cdot x=$If$0$$f_{}(x)=0.5$
    2.$f_{}The\; value\; range\; of\; (x)$ is (0-1)
  . The second feature makes the sigmoid function very suitable as a probability.
  In the perceptron part, it was mentioned that the discriminant function is used to discriminate the category of the data. In this question, it refers to whether the image is a vertical or horizontal category. For ease of handling, the width is set to$x_1$, the height is set to $x_2$, the horizontal setting is set to 1, and the vertical setting is set to 0(different from the vertical setting of the perceptron, which is -1, which is easy to handle, and can be flexibly set here according to the situation). When using the method of logistic regression, there are similar functions for classification, because it is from the perspective of probability, that is, if 70% of an image is horizontal, then this image should be classified as horizontal. SoDiscriminant/classification function expressionFor example: $$y=\{\begin{array}{rlr}1& & (f_{}(x)\ge 0.5)\\ 0& & \left(f_{}\right(x)<0.5)\end{array}$$
  It can be seen from the image of the sigmoid function that $f_{}\left(x\right)\ge 0.5$ ,$i^T\cdot x\ge 0$ , soDiscriminant/classification function expressionFor example: $$y=\{\begin{array}{rlr}1& & i^T\cdot x\ge 0\\ 0& & i^T\cdot x<0\end{array}$$

5.2 Decision boundary

  Since $f_{}\left(x\right)$ is used as a probability, and thenput unknown data $x$ is the probability of a landscape image as$f_{}(x)$For $$f_{}\left(x\right)=P(and=1|x)$$
  After having a discriminant function, you can bring in specific values ​​to feel it. First randomly determine$\theta$ , if it is the following value:$$i=[\begin{array}{r}{i}_0\\ i_1\\ i_2\end{array}]=[\begin{array}{r}-100\\ 2\\ 1\end{array}]$$$$x=[\begin{array}{r}1\\ x_1\\ x_2\end{array}]$$  $i^T\cdot x=-100+2\cdot x_1+x_2$, if the image is horizontal, then $f_{}(x)\ge 0$ , then$i^T\cdot x\ge 0$ , then$-100+2\cdot x_1+x_2\ge 0$，即$x_2\ge -2\cdot x_1+100$ . The classification line is shown in the figure below.

  It can be seen from the figure that we will$i^T\cdot x=0$ as the boundary to classify the data, a straight line like this is also calleddecision boundary. The classification line cannot completely classify the data correctly because we randomly determine the parameters $The\; value\; of\; \theta$ . Therefore, in order to find the correct parameters, we need to define the objective function of the parameters, and differentiate the objective function to obtain the update function of the parameters.

6. Likelihood function

  The objective function here is different from the error function of the least squares method in regression. The error function is obtained by making the error between the result of the discriminant function and the true value as small as possible. And what is the most ideal result in classification? Consider the following conditional probability:
  $P(and=1|x)$ , which means the probability of classifying data x as horizontal. Therefore, the ideal result is:
    when the image is horizontal, that is, y=1, we hope that$P(and=1|x)$ maximum
    When the image is vertical, that is, y=0, we hope that$P(and=0|x)$ The maximum
  existing data is shown in the table below:

image size	shape	y	probability
80x150	portrait	0	P(y=0 |x)
60x110	portrait	0	P(y=0 |x)
35x130	portrait	0	P(y=0 |x)
80x50	horizontal	1	P(y=1 |x)
160x20	horizontal	1	P(y=1 |x)
125x130	horizontal	1	P(y=1 |x)

  Therefore, for all data, we are seeking the overall probability of all data, that is, the joint probability is the largest. The function is as follows: $$L\left(i\right)=P({and}^{(1)}|x^{(1)})\cdot P({and}^{(2)}|x^{(2)})\cdot \cdot \cdot \cdot P({and}^{(6)}|x^{\left(6\right)})$$  Among them,L ( θ ) L(\theta)LLin$L$$($$\theta$$)$$L$ is taken from the likelihood, also known as the likelihood function.likelihood functionGeneralized form: $$L\left(i\right)=\prod _{i=1}^{n}P({and}^{(i)}=1|x^{(i)}{)}^{y^{(i)}}\cdot P({and}^{\left(i\right)}=0|x^{\left(i\right)}{)}^{1-y^{(i)}}$$Q9：式中$y^{\left(i\right)}$ Sum1$1-y^{}$What does${}^{(}$${}^i$${}^{)}$
mean? A9: The two are indices. For their understanding, consider$y^{(i)}=1$时，则$P({and}^{\left(i\right)}=1|x^{(i)}{)}^{y^{(i)}}\cdot P({and}^{\left(i\right)}=0|x^{\left(i\right)}{)}^{1-y^{\left(i\right)}}=P({and}^{\left(i\right)}=1|x^{\left(i\right)})$.
  now ourobjective functionis the likelihood function. Our goal is to maximize the value of the objective function. We need to differentiate the objective function. If the result function of the differentiation is negative, the value of the parameter is updated to the left; if the result function of the differentiation is positive, the value of the parameter is updated to the right. to find the best parameters. This is exactly the opposite of the direction of updating the parameters to find the minimum value of the objective function in regression.
  But directly differentiating the likelihood function has two problems. One is that all the probabilities are less than 1. When calculating the joint probability, it will get smaller and smaller, and the accuracy of the result will appear. The second is that computers take longer to calculate multiplication than to calculate addition. Therefore, it is necessary to take the logarithm of the likelihood function.

7. Log-likelihood function

Q10: Can the logarithm of the likelihood function be directly taken without affecting the final result?
A10: The logarithmic function is monotonically increasing, so if $L(i_1)<L\left(i_2\right)$，有$logL\left(i_1\right)<logL\left(i_2\right)$ , so that$L\left(\theta \right)$ maximization is to make$logL\left(\theta \right)$ is maximized. Now for$Look\; at\; logL\left(\theta \right)$ deformation.

  The above formula mainly uses the basic properties of logarithms, the fourth line uses the property that the sum of all probabilities = 1, and the fifth line is our preset. (See Section 5.2 Decision Boundary)
  So our objective function is the log likelihood function$$logL\left(\theta \right)\_\_=\sum _{i=1}^n(y^{(i)}log{f}_{}(x^{(i)})+(1-y^{(i)})log(1-f_{}(x^{\left(i\right)})\left)\right)$$  Then it is necessary to differentiate the objective function to find the parameterθ \thetaUpdate expression for$\theta \; .$
  Differentiation of the log-likelihood function:

  let$u=logL\left(\theta \right)$ ,$v=f_{}\left(x\right)$ , as follows:

  Let$z=i^{T}x$, and the differential of the sigmoid function is known, that is,$$\frac{d\sigma (x)}{dx}=\sigma \left(x\right)(1-\sigma (x\left)\right)$$

  Finally, the differential of each part is combined and simplified, as follows:

  Since the maximum value is sought, the parameters move in the same direction as the sign of the differential result. Therefore, the update expression of the parameters is:
$$i_j=i_j+the\sum _{i=1}^n(y^{(i)}-f_{}(x^{(i)}))x_j^{(i)}$$
  In order to be consistent with the parameter update expression of regression, the negative sign in the formula after the learning rate is advanced, as follows: $$i_j=i_j-the\sum _{i=1}^n(f_{}(x^{(i)})-y^{(i)})x_j^{\left(i\right)}$$

8. Linear inseparability

  It can be seen from the previous knowledge that the perceptron model can only be used to solve linearly separable problems, which makes it almost impossible to solve practical problems. The logistic regression method can be used to solve linearly inseparable problems, but in the previous example, for ease of understanding, we still use the linearly separable problem of dividing images into horizontal and vertical categories. For the figure below:

  For the classification of the figure above, it is obviously not a linearly separable problem. So you can increase the training data $x_1^2$. Similar to polynomial regression in Regression 1, increase the degree of polynomial. Vector $\theta$ and$x$如下：
$$i=[\begin{array}{r}{i}_0\\ i_1\\ i_2\\ i_3\end{array}]x=[\begin{array}{r}{x}_1\\ x_2\\ x_3\\ x_1^2\end{array}]$$  $i^{}$Define${}^T$$x$$$i^{T}x=i_0{x}_0+i_1{x}_1+i_2{x}_2+i_3{x}_1^2$$  Now for θ \thetaAssign specific values ​​to$\theta$$$i=\left[\begin{array}{r}0\\ 0\\ 1\\ -1\end{array}\right]$$
  $i^{The\; expression\; of\; T}x$is$i^{T}x=x_2-x_1^2$. Then the decision boundary is $i^{T}x=x_2-x_1^2=0$ , that is,$x_2=x_1^2$, as shown in the figure below:
  As can be seen from the above figure, the decision boundary becomes a curve. Since the parameters are randomly set, the data cannot be completely classified. The parameters can be determined using the stochastic gradient descent method and the most value descent method in regression. The higher the degree of the term, the more complex curves can be simulated to solve more complex problems.