Machine learning - SVM kernel function

The kernel function, the principle is relatively simple to understand

But there are still some things I don't understand

For non-linearly separable data, in the current dimension, it is indistinguishable to directly use SVM

Then you can directly upgrade from the current dimension to a higher dimension for calculation.

For example, the original data only has 3 influencing factors $x_a,x_b,x_c$(gender) (age) (appearance), then based on the data of these three dimensions, it can be expanded into a higher dimension (more unnamed influencing factors), such as xc, xd, xe, xf, xg, and then substituted into Computing in the dual problem of SVM

In fact, there is a book "Mathematical Foundations of Artificial Intelligence", the description is quite clear

In addition, when talking about the Gaussian kernel function, the derivation is also given

The derivation here is actually to confirm how the Gaussian kernel function is expanded into a high-dimensional vector, and how the dot product of the high-dimensional vector is calculated using the current low-dimensional data.

$K(x,y)=e^{-||x-y|{|}^2}=e^{-x^{2e}}{\_}^{-y^2}[1+\frac{(2x.y{)}^1}{1!}+\frac{(2x.y{)}^2}{2!}+\frac{(2x.y{)}^3}{3!}...]$

Here, the dot product of high-dimensional vectors is reflected in $e^{-x^{2e}}{\_}^{-y^2}[1+\frac{(2x.y{)}^1}{1!}+\frac{(2x.y{)}^2}{2!}+\frac{(2x.y{)}^3}{3!}...]$

In the SVM dual function solution, it is necessary to solve the dot product

Suppose the original data x is divided into 3 influencing factors $x_a,x_b,x_c$

Then the two data are $(x_a，x_b，x_c)$,$(x_a，x_b，x_{c2})$

Then their dot product is $x_a{x}_a+x_b{x}_b+x_c{x}_{c2}$

But if, now, the three influencing factors are raised to infinite influencing factors through a certain relationship, $x_e,x_f,x_g,x_h,x_k$

These two data become
$x_{and},x_f,x_g,x_h,x_k...$
$x_{e2},x_{f2},x_{g2},x_h,x_{k2}...$

Then their dot product is
$x_{and}{x}_{e2}+x_f{x}_{f2}+x_g+x_h{x}_h+x_k{x}_{k2}+...$

However, because the high-dimensional dot product calculation is too large, find an original low-dimensional calculation formula that is equal to the high-dimensional dot product result, and that calculation formula is the kernel function!

e − x 2 e − y 2 in the Gaussian kernel function [ 1 + ( 2 x . y ) 1 1 ! + ( 2 x . y ) 2 2 ! + ( 2 x . y ) 3 3 ! . . . ] e $e^{-x^{2e}}{\_}^{-y^2}[1+\frac{(2x.y{)}^1}{1!}+\frac{(2x.y{)}^2}{2!}+\frac{(2x.y{)}^3}{3!}...]$ The dot product of two infinite-dimensional data, the corresponding formula is$e^{-||x-y|{|}^2}$ , which is exactly the kernel function

It's just that the Gaussian kernel function also has a parameter σ, also called the kernel radius

$e^{\frac{-||x-y|{|}^2}{2p}}$, when the difference between the two data of x and y is very small and very close, the value of the kernel function is close to 1, and
when the difference between the two data of x and y is very large, the value of the kernel function is close to 0

This is like a normal distribution (Gaussian distribution)

Then the role of σ is for adjustment:

In extreme terms, when the value of σ is very, very, very, very large, if the data gap between x and y is relatively large, the value of the kernel function is still close to 1

Therefore, the kernel function is a low-dimensional and high-dimensional calculation formula for solving linear relationships and replacing high-dimensional dot products with low-dimensional data calculations.