[Support Vector Machine] SVM linearly separable support vector machine learning algorithm - hard margin maximization support vector machine and detailed explanation of examples

Support vector machines (SVM) is a two-class classification model. Its basic model is a linear classifier defined with the largest margin on the feature space. Includes linearly separable support vector machines, linear support vector machines, and nonlinear support vector machines.

When the training data is linearly separable, a linear classifier is learned through hard margin maximization, which is a linearly separable support vector machine, also known as a hard margin support vector machine.

Linearly separable support vector machine learning algorithm

Input: Linearly separable training data set $T=\{(x_1,y_1),(x_2,y_2),\cdots,(x_N,y_N)\}$ , where $x_i\in$ ${\cal X}={\mathbf{R}}^{n}\:,\quad{\gamma_{i}}\in{\cal Y}=\{-1,+1\}\:,\quad i=1,2,\cdots,N\:;$

Output: Maximum Margin Separating Hyperplane and Classification Decision Function

1) Construct and solve constrained optimization problems

$\begin{array}{ll}{\min_{w,b}}&{\frac{1}{2}\parallel w\parallel^{2}}\\{\mathrm{s.t.}}&{y_{i}(w{\bullet}x_{i}+b)-1\geqslant0,\quad i=1,2,\cdots,N}\\\end{array}.$

get optimal solution $w^{*},b^{*}$

Minimize vector norm with constraints

2) Substitute the optimal solution,

Get the separating hyperplane:

$w^{*}\cdot x+b^{*}=0$

Classification decision function:

$f(x)=\mathrm{sign}(w^{*}\cdot x+b^{*})$

example

Training data set: positive example points $x_{1}=(3,3)^{\mathrm{T}},\quad x_{2}=(4,3)^{\mathrm{T}}$ , negative example points $x_{3}=(1,1)^{\mathrm{T}}$ , finding the maximum separation hyperplane, classification decision function and support vector

untie:

1) Construct and solve constrained optimization problems

$\begin{aligned} &\operatorname*{min}_{w,b} \frac{1}{2}({w_{1}}^{2}+{w_{2}}^{2}) \\\\&\mathbf{s.t.} \\ &\mathbf{} 3w_{1}+3w_{2}+b\geqslant1 \\ &4w_{1}+3w_{2}+b\geqslant1 \\ &-w_{1}-w_{2}-b\geqslant1 \end{aligned}$

get optimal solution $w_{1}=w_{2}=\frac{1}{2},\quad b=-2$

Solving optimization problems requires reducing the number of variables

2) Substitute the optimal solution,

Get the separating hyperplane:

$\frac{1}{2}x^{(1)}+\frac{1}{2}x^{(2)}-2=0$

Classification decision function:

$f(x)=\mathrm{sign}(\frac{1}{2}x^{(1)}+\frac{1}{2}x^{(2)}-2)$

Support vector: $x_{1}=(3,3)^{\mathrm{T}}$ , $x_{3}=(1,1)^{\mathrm{T}}$

The support vector is the point at which the equality sign of the constraint condition is established, that is, $y_{i}(w{\bullet}x_{i}+b)-1= 0$ the point at which it is satisfied