[Statistical Learning｜Book Reading] Chapter 7 Support Vector Machine p95-p133

train of thought

Support vector machine is a two-category model. Its basic model is a linear classifier with the largest interval defined in the feature space. The largest interval makes it different from the perceptron. Support vector machines also include kernel techniques, which make them essentially non-linear classifiers. The learning strategy of the support vector machine is to maximize the interval, which can be formalized as a solution to a convex quadratic programming problem, which is also equivalent to the problem of minimizing the regularized hinge loss function. The learning algorithm of support vector machine is an optimization algorithm for solving convex quadratic programming.

---

---

When the input space is a Euclidean space or a discrete set, and the feature space is a Hilbert space, the kernel function represents the inner product between the feature vectors obtained by mapping the input from the input space to the feature space. Nonlinear support vector machines can be learned by using kernel functions, equivalent to implicitlyLearning Linear Support Vector Machines in High-Dimensional Feature Spaces, such methods are called kernel tricks.

Fast Learning Algorithms for Support Vector Machines and Kernel Functions - Sequential Minimal Optimization Algorithm (SMO)

Support Vector Machines

The simplest case of a support vector machine is a linearly separable support vector machine, or a hard-margin support vector machine. The condition for constructing it is that the training data is linearly separable, and its learning strategy ismaximum interval method. It can be expressed as a convex quadratic programming problem, and its original optimization problem is: $$\underset{w,b}{\min }\frac{1}{2}{\Vert w\Vert }^2$$
$$s.t.y_i(w\ast x_i+b)-1\ge 0,i=1,2,...,N$$
obtains the solution to the optimization problem as$w^{\ast }$,$b^{\ast }$ , to obtain a linearly separable support vector machine, the classification hyperplane is$$w^{\ast }\ast x+b^{\ast }=0$$
classification decision function is$$f(x)=sign(w^{\ast }\ast x+b^{\ast })$$

In the maximum interval method, functional interval and geometric interval are important concepts.

The optimal solution of linearly separable support vector machine exists and is unique, and the instance points on the boundary of the interval are support vectors, and the optimal separating hyperplane is completely determined by the support vectors.

The dual problem of the quadratic programming problem is: $$min\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{a}_i{a}_j{y}_i{y}_j(x_i\ast x_j)-\sum _{i=1}^N{a}_i$$
$$s.t.\sum _{i=1}^N{a}_i{y}_i=0$$
$$a_i\ge 0,i=1,2,...,N$$

Usually, linear separable support vector machines are learned by solving the dual problem, that is, firstly solve the optimal value of the dual problem $a^{\ast }$ , and then solve the optimal value$w^{\ast }$ and$b^{\ast }$ , get the separation hyperplane and classification decision function

Linear Support Vector Machines

In reality, the training data is rarely linearly separable, and the training data is often approximately linearly separable. In this case, a linear support vector machine or a soft margin support vector machine is used. Linear support vector machine is the most basic support vector machine.
For noise or exceptions, by introducing slack variables $X_i$, make it 'separable', and get the convex quadratic programming problem learned by linear support vector machine, the original optimal problem is: $$\underset{w,b,x}{\min }\frac{1}{2}{\Vert w\Vert }^2+C\sum _{i=1}^N{X}_i$$
$$s.t.y_i(w\cdot x_i+b)\ge 1-X_i,i=1,2,...,N$$
$$X_i\ge 0,i=1,2,...,N$$
obtains the solution to the optimization problem as$w^{\ast }$,$b^{\ast }$ , to obtain a linearly separable support vector machine, the classification hyperplane is$$w^{\ast }\ast x+b^{\ast }=0$$
classification decision function is$$f(x)=sign(w^{\ast }\ast x+b^{\ast })$$
The solution of linearly separable support vector machine$w^{\ast }$ Unique, but$b^{\ast }$不孪动。
对做 problem is:$$min\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{a}_i{a}_j{y}_i{y}_j(x_i\ast x_j)-\sum _{i=1}^N{a}_i$$
$$s.t.\sum _{i=1}^N{a}_i{y}_i=0$$
$$0\le a_i\le C,i=1,2,...,The\; dual\; learning\; method\; of\; N-$$
linear support vector machine, first solve the dual problem to get the optimal solution$a^{\ast }$ , and then find the optimal solution to the original problem$w^{\ast }$,$b^{\ast }$ , get the separating hyperplane and classification decision function.
The solution of the dual problem$a^{\ast }$ Satisfies$a_i^{\ast }\ge$The instance pointxi x_i$of\; 0$$x_i$called support vectors. The support vector can be on the margin boundary, or between the margin boundary and the separating hyperplane, or on the misclassified side of the separating hyperplane. The optimal separating hyperplane is fully determined by the support vectors.
Linear SVM learning is equivalent to minimizing the second-order norm regularized hinge function $$\sum _{i=1}^N[1-y_i(w\cdot x_i+b){]}_{+}+\lambda ||w|{|}^2$$

Nonlinear Support Vector Machines

For nonlinear problems in the input space, it can be transformed into a linear classification problem in a high-dimensional feature space through nonlinear transformation, and a linear support vector machine can be learned in the high-dimensional feature space. Since in the dual problem of linear support vector machines, both the objective function and the classification decision function only involve the inner product between instances, there is no need to explicitly formulate nonlinear transformation, but the inner product is replaced by the kernel function .

The kernel function represents the inner product between two instances after a nonlinear transformation. Specifically, $K(x,z)$ is a kernel function, or a positive definite sum, which means that there exists an input space$\chi$ to feature space$\Re$ mapping.

Therefore, in the dual problem of linear support vector machine learning, use the kernel function $K(x,z)$ instead of the inner product, the solution is the nonlinear support vector machine$$f(x)=sign(\sum _{i=1}^N{a}_i^{\ast }{y}_{i}K(x,x_i)+b^{\ast })$$

SMO algorithm

$sequentialminimaloptimization,SMO$ algorithm is a fast algorithm for support vector machine learning, which is characterized by continuously decomposing the original quadratic programming problem into quadratic programming subproblems with only two variables, and The problem is solved analytically until all variables satisfy the KKT conditions. In this way, the optimal solution of the original quadratic programming problem is obtained through the heuristic method. Because the sub-problem has an analytical solution, the calculation of the sub-problem is very fast each time. Although the calculation of the sub-problem is many times, it is still efficient overall.