核方法

核方法$K(x,z)=\phi(x)^T\phi(z)$

通常来说，直接将$x$，$z$代入$K(x,z)$：$\mathbb{R}^n \times \mathbb{R}^n \rightarrow \mathbb{R}$计算，复杂度较低

而先把$x$，$z$扩展成更高维的向量$\phi(x)$：$\mathbb{R}^n \rightarrow \mathbb{R}^d$，$\phi(z)$：$\mathbb{R}^n \rightarrow \mathbb{R}^d$，再对它们求内积$\phi(x)^T\phi(z)$：$\mathbb{R}^d \times \mathbb{R}^d \rightarrow \mathbb{R}$，计算量会很大

$K(x,z)$相当于跳过了$\phi(x)$和$\phi(z)$，隐式地定义了$\phi(x)$，如果你仍然想知道$K(x,z)$对应的$\phi(x)$是什么，可以由$K(x,z)$的定义出发一步一步推导出来，但对应的$\phi(x)$可能不唯一

下面举例进行说明

假设$x$，$z \in \mathbb{R}^n$，考虑核函数$K(x,z) = (x^Tz)^2$，现在需要推导出该核函数所对应的高维向量映射$\phi(x)$（推导的思想：把$x_iz_i$混杂的项分开）

$\begin{aligned} K(x,z) &= (x^Tz)^2 \\ &= \left ( \sum_{i=1}^{n}x_iz_i \right )^2 \\ &= \left ( \sum_{i=1}^{n}x_iz_i \right ) \left ( \sum_{j=1}^{n}x_jz_j \right ) \\ &= \sum_{i=1}^{n}\sum_{j=1}^{n}x_ix_jz_iz_j \\ &= \sum_{i,j=1}^{n}(x_ix_j)(z_iz_j) \\ &= \phi(x)^T\phi(z) \end{aligned}$

当$n=3$时，对$x$的映射$\phi(x)$如下所示
$\phi(x) = \begin{bmatrix} x_1x_1\\ x_1x_2\\ x_1x_3\\ x_2x_1\\ x_2x_2\\ x_2x_3\\ x_3x_1\\ x_3x_2\\ x_3x_3 \end{bmatrix}$

可以看出，由$x$，$z$计算$\phi(x)$，$\phi(x)$各自需要$O(n^2)$的时间复杂度，计算$\phi(x)^T\phi(z)$时，由于$\phi(x)$，$\phi(z)$的维度均为$n^2$，故需要$O(n^2)$的时间复杂度，总的来说，计算$\phi(x)^T\phi(z)$需要$O(n^2)$的时间复杂度

然而，计算$K(x,z) = (x^Tz)^2$只需要$O(n)$的时间复杂度

需要注意的是，给定$K(x,z)$，对应的$\phi(x)$不唯一

如$x$，$z \in \mathbb{R}^n$，$K(x,z) = (x^Tz)^2$

一种可能的$\phi(x) = [ x_1x_1, x_1x_2, x_2x_1, x_2x_2 ]^T$

另一种可能的$\phi(x) = [ x_1x_1, \sqrt{2}x_1x_2, x_2x_2 ]^T$

高斯核函数$\begin{aligned}K(x,z) = \exp \left ( -\frac{\left \| x-z \right \|^2}{2\sigma} \right )\end{aligned}$

当$x$和$z$相距很近时，$K(x,z)$趋近于$1$

当$x$和$z$相距很远时，$K(x,z)$趋近于$0$

对于$m$个样本$\left \{ x^{(1)}, x^{(2)}, ... x^{(m)} \right \}$，定义$m \times m$的Kernel Matrix $K$，其中$K_{ij} = K(x^{(i)}, x^{(j)})$

然而并不是每一个$m \times m$的矩阵有资格称为Kernel Matrix，最基本的条件必须为对阵矩阵

因为$K_{ij} = K(x^{(i)}, x^{(j)}) = \phi(x^{(i)})^T \phi(x^{(j)}) = \phi(x^{(j)})^T \phi(x^{(i)}) = K(x^{(j)}, x^{(i)}) = K_{ji}$

此外，对于任意的向量$z$

$\begin{aligned}z^TKz = \sum\limits_{i=0}^{m}\sum\limits_{j=0}^{m}z_iK_{ij}z_j \geqslant 0\end{aligned}$

即Kernel Matrix必须是半正定的

推导如下

$\begin{aligned}&\quad z^TKz\\&=\begin{bmatrix}z_1 & z_2 & \cdots & z_m\end{bmatrix}\begin{bmatrix}K_{11} & K_{12} & \cdots & K_{1m}\\ K_{21} & K_{22} & \cdots & K_{2m}\\ \vdots & \vdots & & \vdots\\ K_{m1} & K_{m2} & \cdots & K_{mm}\end{bmatrix}\begin{bmatrix}z_1\\ z_2\\ \vdots\\ z_m\end{bmatrix}\\&=\begin{bmatrix}\sum\limits_{i=1}^{m}z_iK_{i1} & \sum\limits_{i=1}^{m}z_iK_{i2} & \cdots & \sum\limits_{i=1}^{m}z_iK_{im}\end{bmatrix}\begin{bmatrix}z_1\\ z_2\\ \vdots\\ z_m\end{bmatrix}\\&=z_1\sum\limits_{i=1}^{m}z_iK_{i1} + z_2\sum\limits_{i=1}^{m}z_iK_{i2} + \cdots + z_m\sum\limits_{i=1}^{m}z_iK_{im}\\&=\sum\limits_{i=1}^{m}z_iK_{i1}z_1 + \sum\limits_{i=1}^{m}z_iK_{i2}z_2 + \cdots + \sum\limits_{i=1}^{m}z_iK_{im}z_m\\&=\sum\limits_{j=1}^{m}\sum\limits_{i=1}^{m}z_iK_{ij}z_j\\&=\sum\limits_{i=1}^{m}\sum\limits_{j=1}^{m}z_iK_{ij}z_j\\&=\sum\limits_{i=1}^{m}\sum\limits_{j=1}^{m}z_i\phi\left ( x^{(i)} \right )^T\phi\left ( x^{(j)} \right )z_j\\&=\sum\limits_{i=1}^{m}\sum\limits_{j=1}^{m}z_i \left ( \sum\limits_{k=1}^{d}\phi_k\left ( x^{(i)} \right )\phi_k\left ( x^{(j)} \right ) \right ) z_j\\&=\sum\limits_{k=1}^{d}\sum\limits_{i=1}^{m}\sum\limits_{j=1}^{m}z_i\phi_k\left ( x^{(i)} \right )\phi_k\left ( x^{(j)} \right )z_j\end{aligned}$

对于$\begin{aligned}\sum\limits_{i=1}^{m}\sum\limits_{j=1}^{m}z_i\phi_k\left (x^{(i)}\right )\phi_k\left (x^{(j)}\right )z_j\end{aligned}$，以$m=3$为例进行后续推导

$\begin{aligned}&\sum\limits_{i=1}^{3}\sum\limits_{j=1}^{3}z_i\phi_k\left (x^{(i)}\right )\phi_k\left (x^{(j)}\right )z_j= \\& \quad z_1\phi_k\left (x^{(1)}\right )\phi_k\left (x^{(1)}\right )z_1 + z_1\phi_k\left (x^{(1)}\right )\phi_k\left (x^{(2)}\right ) z_2 + z_1\phi_k\left (x^{(1)}\right )\phi_k\left (x^{(3)}\right )z_3 \\&+z_2\phi_k\left (x^{(2)}\right )\phi_k\left (x^{(1)}\right )z_1 + z_2\phi_k\left (x^{(2)}\right )\phi_k\left (x^{(2)}\right )z_2 + z_2\phi_k\left (x^{(2)}\right )\phi_k\left (x^{(3)}\right )z_3 \\&+z_3\phi_k\left (x^{(3)}\right )\phi_k\left (x^{(1)}\right )z_1 + z_3\phi_k\left (x^{(3)}\right )\phi_k\left (x^{(2)}\right )z_2 + z_3\phi_k\left (x^{(3)}\right )\phi_k\left (x^{(3)}\right )z_3 \\&=\left ( z_1\phi_k\left (x^{(1)}\right ) + z_2\phi_k\left (x^{(2)}\right ) + z_3\phi_k\left (x^{(3)}\right ) \right )^2 \text{（3项之和的完全平方公式）}\\&=\left ( \sum\limits_{i=1}^{3}z_i\phi_k\left(x^{(i)}\right) \right )^2\end{aligned}$

故有
$\begin{aligned}z^TKz=\sum\limits_{k=1}^{d}\sum\limits_{i=1}^{m}\sum\limits_{j=1}^{m}z_i\phi_k\left ( x^{(i)} \right )\phi_k\left ( x^{(j)} \right )z_j=\sum\limits_{k=1}^{d}\left ( \sum\limits_{i=1}^{m}z_i\phi_k\left (x^{(i)}\right ) \right )^2 \geqslant 0\end{aligned}$

更正式的表达为Mercer定理
$K$：$\mathbb{R}^n \times \mathbb{R}^n \rightarrow \mathbb{R}$，$K$是一个合法的核函数的充分必要条件为对于任意$m$个样本$\begin{aligned}\left \{ x^{(1)}, x^{(2)}, ... x^{(m)} \right \}\end{aligned}$，$K$所对应的Kernel Matrix对称且半正定