Support Vector Machine Regression (SVR)

Support vector machine regression (SVR) is an application model of support vector machine in regression problems. There are many variants of SVM regression models based on different loss functions . This article only describes the$\epsilon$SVR model for insensitive loss function.

main idea

Find a separating hyperplane (hypersurface) that minimizes the expected risk.

$\epsilon$-SVR

$\epsilon$- loss function

$\epsilon$- loss function, which is when the error is less than$\epsilon$, the error can be ignored. Conversely, the error is$\xi-|\epsilon|$. As shown in the figure:

based on$\epsilon$- The SVR of the loss function is called$\epsilon$-SVR.
The optimization problem is as follows:

$\nu$-SVR

same$\nu$- Support vector machine classification, set another parameter$\nu$to adjust the number of support vectors.
The optimization problem is as follows:
$C,\nu$It is set freely by the user, so directly set the$C/l$called$C$,$C\nu$called$\nu$, the optimization function is equivalent to:

$$\min_{w, b, \xi^*,\xi, \epsilon}\qquad \frac{1}{2}w^Tw+\nu\epsilon+C\sum_{i=1}^{l}(\xi_i+\xi_i^*)$$

support vector

Intuitively, the support vector is the sample that plays a role in the calculation of the final w and b ($\alpha\geqslant0$). Then according to$\epsilon$Insensitive function image, the insensitive area is like a "pipe". Sample correspondence within the pipeline$\alpha=0$, is the non-support vector; the one located on the "tube wall" is the boundary support vector,$0<\alpha<C$, is the boundary support vector; outside the "pipe" is the non-boundary support vector,$\alpha\geqslant C$, is a non-boundary support vector (outliers are often selected from non-boundary support vectors during outlier detection);
Note: The picture is from the LIBSVM guidance document . Please correct me if I am wrong.