Kernel (kernel function)

1.kernel introduction

There are two common problems with machine learning: 1. Weighting. 2. Find the similarity (distance).

Generally speaking, the higher the similarity, the higher the weight, but how to find it?
You can use
1.k nearest neighbors (the closer the distance, the greater the weight)
2. Nadaraya-Watson estimation (the farther the distance, the greater the weight)
f(x ) = w ^T y
where w is wi _i = К(x _i ,μ)
where К(x _i ,μ) is the kernel function, which is also called similar function here. That is a way to find distance. This is the distance between xi and other x.

Talk about several commonly used kernel forms:
1. Basic Gaussian kernel:

2. If ∑ is a diagonal matrix, it can be written as:

3. If the values ​​on the diagonal matrix are all the same, then it can be written as:

4.
This is often used in NLP, xi represents a vector in it, and the element xij in the vector represents the number of times the j-th word appears in article i, which is actually not good. Many words have no discriminative degree, and then we used TF-IDF (an optimization of the above algorithm) to do this.

2. How to learn kernel

-Maximizing likelihood (maximum likelihood estimation)
-MKL (multiple kernel learning)
К(x,x ^' ) = ∑ _j w _j К _j (x,x ^' )
К _j (x,x ^' ) is the jth small For the kernel, what we want to use is the weighted result К(x,x ^' ).
-Adaptive basis function model (ABM) In

this formula, we have to learn w ₀ and Φ.

3. Code

# -*- coding: utf-8 -*-
"""
Created on Thu Nov 19 22:26:36 2020

@author: 13056
"""
from sklearn import svm,datasets
import numpy as np
import matplotlib.pyplot as plt

iris = datasets.load_iris();
print("iris的类型",type(iris))
print('iris的属性',iris.keys())

X = iris.data[:,:2]
#拿二维属性
Y = iris.target

# create instances of svm
linear = svm.SVC(kernel = 'linear').fit(X,Y)#W'x
rbf = svm.SVC(kernel = 'rbf', gamma = 0.7).fit(X,Y)
poly = svm.SVC(kernel = 'poly', degree = 3).fit(X,Y)

# create a mesh
x0_min, x0_max = X[:,0].min()-0.1, X[:,0].max()+0.1
x1_min, x1_max = X[:,1].min()-0.1, X[:,1].max()+0.1
h = .02 # step size in the mesh
xx,yy = np.meshgrid(np.arange(x0_min,x0_max,h),np.arange(x1_min,x1_max,h))

titles = ['linear','rbf','poly']
for i,clf in enumerate((linear,rbf,poly)):
    plt.subplot(1,3,i+1)
    plt.subplots_adjust(wspace = 0.4, hspace = 0.4)
    Z = clf.predict(np.c_[xx.ravel(),yy.ravel()])
    Z = Z.reshape(xx.shape)
    #Z = np.random.randint(0,3,xx.shape)
    
    #draw decision boundary
    plt.contourf(xx,yy,Z,cmap = plt.cm.cool)
    plt.scatter(X[:,0],X[:,1],c = Y, cmap = plt.cm.cool)
    plt.title(titles[i])
plt.show()

Result:
Different colors are of different types.