[Machine Learning] Support Vector Machine SVM

The support vector machine involves several difficult problems such as linear programming, convex optimization, and matrix analysis. This is my third time to learn it. Now it seems that there must be a fourth and fifth time in the future. Machine learning and deep learning are basically separated from support vector machines, but the ideas in them are still worth pondering and pondering over and over again. Studying this kind of thing also reminds me of the importance of mathematical ability on the academic road.

(There may be many questions in this article, I hope you can correct me.)

1. Prior knowledge of mathematics

1.1. Distance from point to plane

The discussion here is not only two-dimensional, but also high-dimensional:

First consider how a plane is represented, assuming that the unit normal vector of a plane is $\omega$ , then any point on this plane should satisfy the inner product of the normal vector and the distance h from the origin to the hyperplane (because the vector connected by a point on the plane and the origin can be split into the direction along the normal vector and the direction perpendicular to the normal vector), So it can be expressed as:
$$\omega x=h$$
is usually expressed as:
$$\omega x+b=0$$
then any point$x_i$The distance to this plane is:
$$d=\omega x_i-h=\omega x_i+b$$
Of course, we assumed that the normal vector is a unit normal vector. If it is not a unit normal vector, the distance should be as follows:
$$d=\frac{\omega x_i+b}{||\omega ||}$$
Note that the above distance can be positive or negative, and the plus or minus sign indicates which side of the hyperplane it is on.

1.2. Lagrange multiplier method

The Lagrange multiplier method is the knowledge learned in Calculus 2, which is used to solve extreme value problems under constraints.

Let’s first look at the solution to extreme value problems with equality constraints, such as the following optimization problem:
$$\begin{array}{c}{\min }_{w}f(w)\\ \text{ s.t. }{h}_i(w)=0,i=1,\dots ,l\end{array}$$
The objective function is f(w) with the following equality constraints. Usually the solution is to introduce the Lagrangian operator, here use $\beta$ is used to represent the operator, and the Lagrange formula is
$$\mathcal{L}(w,b)=f\left(w\right)+\sum _{i=1}^l{b}_i{h}_i\left(w\right)$$
Next, only the partial derivative is required to solve w.

Wait for the opportunity to add it here.

2. Core idea:

2.1. Thought

Find the hyperplane for classification, so that the point closest to the hyperplane is as far away from the hyperplane as possible .

This idea seems very simple, but it uses many optimization methods and techniques in actual operation, which is quite difficult.

3. Hard spacing problem:

3.1. Optimization objectives and basic deformations

Well, let's find a way to divide the data into the left and right sides of the plane, and the data value is $x_i$, labeled as $y_i$, the distance between the data and the plane is $d_i$(Again, d can be positive or negative). We think that the points on the hyperplane d>0 should satisfy y=1, and the points on d<0 should satisfy y=-1. Then $d\ast y$ represents the absolute value of the distance.

Let the hyperplane be $w^{T}x+b=0$ , then our core idea (makingthe point closest to the hyperplaneas far awayfrom the hyperplane) can be expressed as follows

$$\max \underset{i}{\min }{c}_i=\frac{y_i\ast \left(w^T{x}_i+b\right)}{\Vert w{\Vert }_2}$$
问题可以描述为：
max ⁡ min ⁡ γ i = y i ∗ ( w x i + b ) ∥ w ∥ 2  s.t.  y i [ ( x i ⋅ w ) + b ] ≥ d m i n , i = 1 , 2 , ⋯   , l  for  ( y 1 , x 1 ) , ⋯   , ( y l , x l ) , y ∈ { − 1 , 1 } \begin{array}{l} \max \min \gamma_i=\frac{y_i *\left(w x_i+b\right)}{\|w\|_{2}}\\ \text { s.t. } \quad y_{i}\left[\left(\mathbf{x}_{i} \cdot \mathbf{w}\right)+b\right] \geq d_{min}, \quad i=1,2, \cdots, l \\ \text { for } \quad\left(y_{1}, \mathbf{x}_{1}\right), \cdots,\left(y_{l}, \mathbf{x}_{l}\right), y \in\{-1,1\} \\ \end{array} maxminci​=∥w∥2​yi​∗(wxi​+b)​ s.t. yi​[(xi​⋅w)+b]≥dmin​,i=1,2,⋯,l for (y1​,x1​),⋯,(yl​,xl​),y∈{ −1,1}​

General $d_{min}$divide to one side to get $w^{\prime }$$b^{\prime }$,可以重写如下的约束：
max ⁡ min ⁡ γ i = y i ∗ ( w ′ x i + b ′ ) ∥ w ′ ∥ 2  s.t.  y i [ ( x i ⋅ w ′ ) + b ′ ] ≥ 1 , i = 1 , 2 , ⋯   , l  for  ( y 1 , x 1 ) , ⋯   , ( y l , x l ) , y ∈ { − 1 , 1 } \begin{array}{l} \max \min \gamma_i=\frac{y_i *\left(w' x_i+b'\right)}{\|w'\|_{2}}\\ \text { s.t. } \quad y_{i}\left[\left(\mathbf{x}_{i} \cdot \mathbf{w'}\right)+b'\right] \geq 1, \quad i=1,2, \cdots, l \\ \text { for } \quad\left(y_{1}, \mathbf{x}_{1}\right), \cdots,\left(y_{l}, \mathbf{x}_{l}\right), y \in\{-1,1\} \\ \end{array} maxminci​=∥w′∥2​yi​∗(w′xi​+b′)​ s.t. yi​[(xi​⋅w′)+b′]≥1,i=1,2,⋯,l for (y1​,x1​),⋯,(yl​,xl​),y∈{ −1,1}​
Simple considerations can be imagined, there must be an inequality at the boundary value that becomes an equality constraint, and since min is for i, and $The\; denominator\; of\; \gamma$ is irrelevant, so the minimum value of the numerator is constrained to be 1 by the equation, and the purpose of the problem becomes to minimize the denominator:

max ⁡ 1 ∥ w ′ ∥ 2  s.t.  y i [ ( x i ⋅ w ′ ) + b ′ ] ≥ 1 , i = 1 , 2 , ⋯   , l  for  ( y 1 , x 1 ) , ⋯   , ( y l , x l ) , y ∈ { − 1 , 1 } \begin{array}{c} \max \frac{1}{\|w'\|_{2}}\\ \text { s.t. } \quad y_{i}\left[\left(\mathbf{x}_{i} \cdot \mathbf{w'}\right)+b'\right] \geq 1, \quad i=1,2, \cdots, l \\ \text { for } \quad\left(y_{1}, \mathbf{x}_{1}\right), \cdots,\left(y_{l}, \mathbf{x}_{l}\right), y \in\{-1,1\} \\ \end{array} max∥w′∥2​1​ s.t. yi​[(xi​⋅w′)+b′]≥1,i=1,2,⋯,l for (y1​,x1​),⋯,(yl​,xl​),y∈{ −1,1}​

通常写成这样：
min ⁡ Φ ( w ) = 1 2 ( w ⋅ w )  w.r.t.  w  s.t.  y i [ ( x i ⋅ w ) + b ] ≥ 1 , i = 1 , 2 , ⋯   , l  for  ( y 1 , x 1 ) , ⋯   , ( y l , x l ) , y ∈ { − 1 , 1 } \begin{array}{l} \min \Phi(\boldsymbol{w})=\frac{1}{2}(\boldsymbol{w} \cdot \boldsymbol{w}) \text { w.r.t. } \boldsymbol{w} \\ \text { s.t. } \quad y_{i}\left[\left(\mathbf{x}_{i} \cdot \mathbf{w}\right)+b\right] \geq 1, \quad i=1,2, \cdots, l \\ \text { for } \quad\left(y_{1}, \mathbf{x}_{1}\right), \cdots,\left(y_{l}, \mathbf{x}_{l}\right), y \in\{-1,1\} \\ \end{array} minΦ ( w )=21​(w⋅w) w.r.t. w s.t. yi​[(xi​⋅w)+b]≥1,i=1,2,⋯,l for (y1​,x1​),⋯,(yl​,xl​),y∈{ −1,1}​

PS: About finding the objective function here, in fact, there is a more intuitive understanding idea in the MIT course (see the MIT course in the reference).

3.2. Lagrangianization of the optimization objective

Now do the equivalent transformation of the constraints: $1-y_i\left(w^T{x}_i+b\right)\le 0$ , and then use the Lagrange daily number method to construct the Lagrangian function:
$$L(w,b,l)=\frac{1}{2}{w}^{T}w+\sum _{i=1}^N{l}_i\left[1-y_i\left(w^T{x}_i+b\right)\right]$$
We want to use$l_i\ge 0$ ,去除去 (w, b) 中使 $y_i\left(w^T{x}_i+b\right)<1$ , the analysis is as follows:

• 若 $1-y_i\left(w^T{x}_i+b\right)>0$ , 则${}_l^{\max }L(w,b,l)=\frac{1}{2}{w}^{T}w+\infty =\infty$
• 若 $1-y_i\left(w^T{x}_i+b\right)\le 0$ , 则${}_l^{\max }L(w,b,l)=\frac{1}{2}{w}^{T}w+0=\frac{1}{2}{w}^{T}w$

从而, ${\min }_{w,b}{\max }_{}L(w,b,l)={\min }_{w,b}\left(\infty ,\frac{1}{2}{w}^{T}w\right)={\min }_{w,b}\frac{1}{2}{w}^{T}w$, the condition$l_i\ge 0$ .

Therefore, the constrained model will be transformed into an unconstrained model for (w, b):
$$\{\begin{array}{cc}{\min }_{w,b}{\max }_{}& L(w,b,l)\\ \text{ s.t }& l_i\ge 1\end{array}$$

3.3. Lagrangian dualization

Usually it is more convenient for us to find the minimization problem, that is, the derivative is 0, so we convert the above Lagrangian function into its dual problem, that is, $\min$ 、 $The\; max$ solution order is reversed and becomes$\max$、$\min$ . There is an obvious conclusion, that is, first find the minimum value for a sequence, and then find a maximum value a among all the minimum values; find the maximum value for a sequence first, and then find a minimum value b among all the maximum values . $a\le b$ Assignment to the following parameters:
$$\underset{a,b,a_i\ge 0}{\max }\underset{x}{\min }L(\mathbf{x},a,b)\le \underset{x}{\min }\underset{a,b,a_i\ge 0}{\max }L(x,a,\beta )$$
This relationship is often called weak duality, and strong duality means taking the equal sign.
Due to a series of properties of this Lagrangian function, it can be proved that the last functional relation is actually a strong dual. (Maybe only after I learn convex optimization and come back to understand slowly, cry and chat)

3.4. Dual problem optimization

First, 分$\lambda$ is regarded as a constant solution,$L(\omega ,b,\lambda )$ takes the minimum value$\left({oh}^{\ast },b^{\ast }\right)$, and then for$\lambda$ to limit, solve$\min L(\omega ,b,\lambda )\_\left({oh}^{\ast },b^{\ast }\right)$ 过程如下:
$$\begin{array}{rl}\frac{\partial L(\omega ,b,l}{\partial b}& =\frac{\partial }{\partial b}\left\{\frac{1}{2}{w}^T\omega +\sum _{i=1}^N{l}_i\left[1-y_i\left(w^T{x}_i+b\right)\right]\right\}\\ & =\frac{\partial }{\partial b}\left(-\sum _{i=1}^N{l}_i{y}_{i}b\right)\\ & =-\sum _{i=1}^N{l}_i{y}_i\\ & =0\end{array}$$

令 $\frac{\partial l}{\partial b}=0$ 有 ${\sum }_{i=1}^N{l}_i{y}_i=0From$ L ( ω , b , λ ) L(\omega ,$L(\omega ,b,l)$

$$\begin{array}{rl}L(\omega ,b,l)& =\frac{1}{2}{w}^T\omega +\sum _{i=1}^N{l}_i\left[1-y_i\left(w^T{x}_i+b\right)\right]\\ & =\frac{1}{2}{w}^T\omega +\sum _{i=1}^N{l}_i-\sum _{i=1}^N{l}_i{y}_i{w}^T{x}_i-\sum _{i=1}^N{l}_i{y}_{i}b\\ & =\frac{1}{2}{w}^T\omega +\sum _{i=1}^N{l}_i-\sum _{i=1}^N{l}_i{y}_i{w}^T{x}_i\\ \frac{\partial L_{(\omega ,b,\lambda }}{\partial \omega }& =\frac{\partial }{\partial \omega }\left[\frac{1}{2}{w}^T\omega +\sum _{i=1}^N{l}_i-\sum _{i=1}^N{l}_i{y}_i{w}^T{x}_i\right]\\ & =\frac{1}{2}\cdot 2o-\sum _{i=1}^N{l}_i{y}_i{x}_i\end{array}$$

令$\frac{\partial l}{\partial \omega }=0$ ω$oh={\sum }_{i=1}^N{l}_i{y}_i{x}_i$Substitute into $l(o,b,\lambda )$ get:
$$\begin{array}{rl}L(\omega ,b,l)& =\frac{1}{2}{\left(\sum _{i=1}^N{l}_i{y}_i{x}_i\right)}^T\left(\sum _{i=1}^N{l}_j{y}_j{x}_j\right)-\sum _{i=1}^N{l}_i{y}_i{\left(\sum _{i=1}^N{l}_j{y}_j{x}_j\right)}^T{x}_i+\sum _{i=1}^N{l}_i\\ & =\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{l}_i{l}_j{y}_i{y}_j{x}_i^T{x}_j-\sum _{i=1}^N\sum _{1j=}^N{l}_i{l}_j{y}_i{y}_j{x}_j^T{x}_i+\sum _{i=}^N{l}_j\\ & =\sum _{i=1}^N{l}_i-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{l}_i{l}_j{y}_i{y}_j{x}_i^T{x}_j\end{array}$$
于是对偶问题优化模型为:
$$\{\begin{array}{c}\max {\sum }_{i=1}^N{l}_i-\frac{1}{2}{\sum }_{i=1}^N{\sum }_{j=1}^N{l}_i{l}_j{y}_i{y}_j{x}_i^T{x}_j\\ st.{\lambda }_i\ge 0,\text{ for }\forall i=1,2,\cdots ,N.\\ {\sum }_{i=1}^N{l}_i{y}_i=0\end{array}$$

4. Soft margin problem

If the data is linearly inseparable, then increase the relaxation factor:
$$X_i\ge 0$$
makes the function interval plus the slack variable greater than or equal to 1, then the constraint condition becomes
$$y_i\left(w{x}_i+b\right)\ge 1-X_i$$
The objective function, the latter term is to make this relaxation (error) as small as possible:
$$\underset{w,b}{\min }\frac{1}{2}\Vert w{\Vert }^2+C\sum _{i=1}^N{X}_i$$
此时的凸优化为
$$\begin{array}{c}{\min }_{w,b,}\frac{1}{2}\Vert w{\Vert }^2+C{\sum }_{i=1}^N{X}_i\\ \text{ s.t. }{y}_i\left(w{x}_i+b\right)\ge 1-X_i,i=1,2,\cdots ,n\\ X_i\ge 0,i=1,2,\cdots ,n\end{array}$$
可以写出如下的拉格朗日函数：
$$\begin{array}{l}L(w,b,x,a,u)=\frac{1}{2}\Vert w{\Vert }^2+C{\sum }_{i=1}^n{X}_i-{\sum }_{i=1}^n{a}_i\left(y_i\left(w{x}_i+b\right)-1+X_i\right)-{\sum }_{i=1}^n{b}_i{X}_i\end{array}$$

Also by derivation we can get the iterative mode and constraints of parameters:
$$\begin{array}{l}\frac{\partial L}{\partial w}=0\Rightarrow w={\sum }_{i=1}^n{a}_i{y}_i{x}_i\\ \frac{\partial L}{\partial b}=0\Rightarrow 0={\sum }_{i=1}^n{a}_i{y}_i\\ \frac{\partial L}{\partial \xi }=0\Rightarrow C-a_i-b_i=0\end{array}$$
由KKT条件：
$$a_i\left(y_i\left(w{x}_i+b\right)-1+X_i\right)=0$$

因为
$$y_i\left(\left(\mathbf{w}\cdot {\mathbf{x}}_i\right)+b\right)\ge 1-X_i$$
Default $a_i\ne 0$ can only be established for samples that are correctly classified and located on the boundary, that is:

$$\begin{array}{l}{y}_i\left(\left(w\cdot x_i\right)+b\right)=1-X_i\\ 0\le a_i\le C,X_i=0\end{array}$$
Misclassified samples
$$a_i=C,X_i>0indefinitely$$ { ∑ i = 1 n ( C − α i − β i ) = 0 α i ≥ 0 β i ≥ 0 \left\{\begin{array}{c} \sum_{i=1}^{
n
$$\{\begin{array}{c}{\sum }_{i=1}^n\left(C-a_i-b_i\right)=0\\ a_i\ge 0\\ b_i\ge 0\end{array}$$
故
$$0\le a_i\le C$$
将
$${\mathbf{w}}_0=\sum _{SVs}{a}_i{y}_i{x}_i,a_i\ge 0Specific$$
voltage range, cycle, range range
$$\begin{array}{rl}\max W\left(a\right)=& \sum _{i=1}^l{a}_i-\frac{1}{2}\sum _{i,j=1}^l{a}_i{a}_j{y}_i{y}_j\left(x_i\cdot x_j\right)\\ & \text{ s.t. }\sum _{i=1}^l{y}_i{a}_i=0\\ 0\le & a_i\le C,i=1,2,\dots ,l\end{array}$$

References:

course:

• ML Lecture 20: Support Vector Machine (SVM) - YouTube
• Whiteboard derivation at station b [Machine Learning - Whiteboard Derivation Series (6) - Support Vector Machine SVM (Support Vector Machine)]
• MIT 16. Learning: Support Vector Machines - YouTube

blog:

• How does the SVM support vector machine get the score?
• SVM (Support Vector Machine) principle and detailed explanation of the whole process of mathematical derivation (with MATLAB program) - Zimu's article - Know almost
• station b notes