Deep Learning - Support Vector Machine (SVM)

1. Introduction
In the field of machine learning, SVM (Support Vector Machine) is a supervised learning model, usually used for pattern recognition, classification (outlier detection) and regression analysis. In the SVM algorithm, we plot the data in an n-dimensional space (n represents the number of features of the data), and then find a hyperplane that can separate the data into two categories. SVM can generally only $It\; is\; used\; in\; binary\; classification\; problems$ and does not work well for multi-class problems.
The learning algorithm of SVM is the optimization algorithm for solving convex quadratic programming. The basic idea of ​​SVM learning$The\; basic\; idea\; is\; to\; solve\; the\; separating\; hyperplane\; that\; can\; correctly\; divide\; the\; training\; data\; set\; and\; have\; the\; largest\; geometric\; interval$ . For linearly separable data sets, there are infinitely many such hyperplanes (ie, perceptrons), but the separating hyperplane with the largest geometric interval is unique.
SVM will look for the dividing hyperplane that can distinguish the two classes and maximize the margin. It is better to divide the hyperplane, the local disturbance of the sample has the least impact on it, the classification results are the most robust, and the generalization ability to unseen examples is the strongest.
For any hyperplane, the data points on both sides of it have a minimum distance (vertical distance) from it, and the sum of the two minimum distances is the interval. For example, the banded area formed by two dotted lines in the figure below is the margin, and the dotted line is determined by the two points closest to the central solid line (that is, determined by the support vector). But at this time, the margin is relatively small. If we use the second method to draw, the margin will become larger and closer to our goal.


2. The difference between SVM and KNN
KNN classification problem:$Determinethe\; Kvalue\; and\; draw\; a\; circle,\; and\; which\; points\; are\; closer\; to\; which\; type$

KNN has no training process. The basic principle is to find the K values ​​​​that are closest to the sample points that need to be predicted in the training data set (distances can be used such as Euclidean distance, The value of K needs to be adjusted by yourself) to achieve classification.
SVM classification problem:$Find\; the\; decision\; boundary\; and\; divide\; the\; data.$
SVM needs a hyperplane wx+b to divide the data set (here, it can be linearly divided into an example), so there will be a model training process to find the values ​​​​of w and b. After the training is completed, it can be used for prediction. The label of the sample point x is determined according to the value of the function y=wx+b, and there is no need to consider the training set. For SVM, a model is first trained on the training set, and then the model is used to directly classify the test set.

Reference:
Deep Learning Algorithm 2-SVM Principle
Deep Learning SVM
Deep Learning - SVM
Deep Learning Lesson 4 - SVM