Support Vector Machines

Source: http://www.hankcs.com/ml/support-vector-machine.html

1. Overview

Support vector machines (SVM) are a two-class classification model.

The learning algorithm of support vector machine is an optimization algorithm for solving convex quadratic programming.

Classification:

Linear support vector machine in linearly separable case

Linear support vector machine

Non-linear support vector machine (non-linear support vector machine).

It is assumed that the input space and the feature space are two different spaces . The input space is Euclidean space or discrete set, and the feature space is Euclidean space or Hilbert space.

Linear separable support vector machine and linear support vector machine assume that the elements of these two spaces correspond one-to-one, and map the input in the input space to the feature vector in the feature space.

A nonlinear support vector machine uses a nonlinear mapping from the input space to the feature space to map the input to feature vectors.

From the input space to the feature space, the learning of the support vector machine is carried out in the feature space.

Given a training data set on a feature space, xi is the i-th feature vector, also called an instance, yi is the class label of xi , and xi is called a positive example or a negative example when it is positive or negative. Finding a separating hyperplane in the space can classify instances into different classes.

2. Linearly separable support vector machine

When the training dataset is linearly separable, there are infinite separation hyperplanes that can correctly separate the two types of data. The perceptron uses the strategy with the least misclassification to obtain the separating hyperplane, but there are infinitely many solutions at this time. The linear separable support vector machine uses the interval maximization to find the optimal separating hyperplane, at this time, the solution is unique.

The basic idea of SVM learning is to solve the separating hyperplane that correctly partitions the training dataset and has the largest geometric separation.

Maximum interval algorithm

Linear Separable Support Vector Machine Learning Algorithm - Maximum Margin Method

support vector

The support vector is the point at which the equal sign of the constraint expression holds

The points on H1 and H2 are the support vectors.

H1 and H2 are parallel and no instance point falls between them. A long band is formed between H1 and H2 with the separating hyperplane parallel to and at their center.

The width of the long strip, that is, the distance between H1 and H2 is called the margin. interval equals .

H1 and H2 are called interval boundaries.

The support vector plays a decisive role in determining the separation hyperplane, so this classification model is called a support vector machine.

Dual Algorithm

propose:

In order to solve the optimization problem of linearly separable support vector machine

, regard it as the original optimization problem, and apply Lagrangian duality to obtain the optimal solution of the original problem by solving the dual problem.

Ideas:

Given a linearly separable training dataset, one can first find the dual problem

The solution of the original problem is obtained by using the formula and the formula ; thus the separation hyperplane and the classification decision function are obtained.

algorithm:

(1) Construct and solve the constrained optimization problem

Find the optimal solution .

(2) Calculation

and choose a positive component , compute

(3) Obtain the separating hyperplane

Classification decision function:

3. Linear Support Vector Machines

Modified hard margin maximization to become soft margin maximization, extending to linear inseparable problems.

Assume that the training dataset is not linearly separable. There are some singular points in the training data. After removing these singular points, the set of most of the remaining sample points is linearly separable.

A loose pool variable is introduced for each sample point so that the function interval plus the slack variable is greater than or equal to 1.

The constraints become

definition

For a given linearly inseparable training dataset, by solving a convex quadratic programming problem

The resulting separating hyperplane is

and the corresponding classification decision function

It is called a linear support vector machine.

Linear Support Vector Machine Algorithm

(1) Select the penalty parameter C>0, construct and solve the convex quadratic programming problem

Find the optimal solution .

(2) Calculation

选择的一个分量适合条件,计算

(3)求得分离超平面

分类决策函数：

注：b的解并不唯一，实际计算时可以取在所有符合条件的样本点上的平均值。

支持向量

在线性不可分的情况下，将对偶问题

的解中对应于的样本点的实例,称为支持向量（软间隔的支持向量)。

四、非线性支持向量机

思想：

对于非线性问题，进行一个非线性变换，将非线性问题变换为线性问题，通过解变换后的线性问题的方法求解原来的非线性问题。

在对偶问题的目标函数中的内积用核函数来代替。

对偶问题的目标函数成为

分类决策函数中的内积也用核函数代替，分类决策函数式成为

定义

从非线性分类训练集，通过核函数与软间隔最大化，或凸二次规划,学习得到的分类决策函数

称为非线性支持向量，是正定核函数。

算法：

选取适当的核函数和适当的参数C,构造并求解最优化问题

五、序列最小最优化SMO

序列最小最优化（sequential minimal optimisation,SMO)算法

SMO算法将原问题不断分解为子问题并对子问题求解，进而达到求解原问题的目的。

SMO算法包括两个部分：求解两个变量二次规划的解析方法和选择变量的启发式方法。

基本思路：

如果所有变量的解都满足此最优化问题的KKT条件，那么这个最优化问题的解就得到了。

否则，选择两个变量，固定其他变量，针对这两个变量构建一个二次规划问题。

子问题有两个变量，一个是违反KKT条件最严重的那一个，另一个由约束条件自动确定。

由于等式约束，子问題的两个变量中只有一个是自由变量。

两个变量二次规划的求解方法

假设选择的两个变量是、，其他变量是固定的。

于是SMO的最优化问题

的子问题可以写成：

其中，，是常数。

公式解：

最优化问题

沿着约束方向未经剪辑时的解是

其中，

是输入空间到特征空间的映射，由式

给出。

变量的选择

SMO称选择第1个变量的过程为外层循环。外层循环在训练样本中选取违反KKT条件最严重的样本点。

首先遍历所有满足条件支持向量机341.png 的样本点，即在间隔边界上的支持向量点，如果这些样本点都满足KKT条件，则遍历整个训练集。

SMO称选择第2个变量的过程为内层循环。选择支持向量机324.png ，使其对应的支持向量机345.png 最大。

每次完成两个变量的优化后，都要重新计算阈值支持向量机348.png 。

If 支持向量机359.png both conditions are met 支持向量机360.png , then 支持向量机361.png . If it 支持向量机362.png is 0 or C, then the 支持向量机363.png sum 支持向量机364.png and the numbers between them are the thresholds that meet the KKT conditions, and their midpoints are selected as 支持向量机365.png .