机器学习算法3_岭回归

机器学习算法第三篇

1. 本文目内容:岭回归算法推导
2. 本文基于多元线性回归
3. 数学核心向
   $\$
   $\$

背景:

标准方程法解线性回归时的代价函数矩阵化过程中
当x_data中数据的特征比数据的行数多时候,
无法化成下式子,因为 $X^TX$不是满秩矩阵,无法求逆
$\theta=(X^TX)^{-1}X^TY$

$\$
$\$

岭回归概念

• 故科学家们在原代价函数中导入了偏差项 $\frac{1}{m}\sum_{j=1}^n\theta^2$,解决该问题
  $\$

$标准方程法代价函数\Rightarrow J(\theta_1,\theta_2...\theta_n)= \frac{1}{m}\sum_{i=1}^m (h_\theta(x^i)-y^i)^2\tag 1$

$岭回归代价函数\Rightarrow J(\theta_1,\theta_2...\theta_n)= \frac{1}{m}\left\{\sum_{i=1}^m (h_\theta(x^i)-y^i)^2+ \lambda\sum_{j=1}^n\theta^2\right\}\tag 2$
$(2)式矩阵化得\Rightarrow \theta=(X^TX+\lambda I)^{-1}X^TY1\tag 3$

岭回归的优点

1. 解决标准方程法中求逆问题(上面已述)
2. 通过调节 $\lambda$的值可以得到更好的估计,(l2正则化分析,本文暂不展开,以后开篇专门分析)
3. 解决多重公线问题(非本文内容暂不展开)

推导过程

• 推导过程指展开上述(2)式到(3)的变换过程

• 首先假设有数据Data( 上标为列号,下标为行号)
  $\begin{bmatrix} x_1^1 & x_2^1 \quad ...&x_n^1& y^1 \\ x_1^2 & x_2^2 \quad...&x_n^2& y^2\\ x_1^3 & x_2^3 \quad...&x_n^3& y^3 \\ . & . \quad...& .& . \\ . & . \quad...& .& . \\ x_1^m & x_2^m \quad...&x_n^m& y^m \\ \end{bmatrix}$
  $令x\_data =\begin{bmatrix} x_1^1 & x_2^1 \quad ...&x_n^1\\ x_1^2 & x_2^2 \quad...&x_n^2\\ x_1^3 & x_2^3 \quad...&x_n^3 \\ . & . \quad...& . \\ . & . \quad...& .\\ x_1^m & x_2^m \quad...&x_n^m\\ \end{bmatrix}$
  $令y\_data =\begin{bmatrix} y^1 \\ y^2\\ y^3 \\ . \\ . \\ y^m \\ \end{bmatrix}$

$\$

$\$

$\$

$\$

• 然后将代价函数(2)中的子式分别转换成矩阵

$\theta= \begin{bmatrix} \theta_1 \\ \theta_2\\ \theta_3 \\ . \\ . \\ \theta_n \\ \end{bmatrix}\tag 4$

$\$

$J(\theta_1,\theta_2...\theta_n) \Rightarrow J(\theta)\tag 5$

$\$

$\sum_{j=1}^n\theta^2=(\theta_1^2+\theta_2^2+\theta_3^2+..+\theta_n^2)=\theta ^T\theta\tag 6$

$\$

$y^i\Rightarrow Y=\begin{bmatrix} y^1 \\ y^2\\ y^3 \\ . \\ . \\ y^m \\ \end{bmatrix}\tag 7$

$\$

$x^i\Rightarrow X=\begin{bmatrix} x_1^1 & x_2^1 \quad ...&x_n^1\\ x_1^2 & x_2^2 \quad...&x_n^2\\ x_1^3 & x_2^3 \quad...&x_n^3\\ . & . \quad...& . \\ . & . \quad...& .\\ x_1^m & x_2^m \quad...&x_n^m\\ \end{bmatrix}\tag 8$

$\$

$h_\theta(x^i)\Rightarrow \begin{bmatrix} h_\theta (x^1) \\ h_\theta (x^2) \\ h_\theta (x^3 ) \\ . \\ . \\ h_\theta (x^m) \\ \end{bmatrix}=\begin{bmatrix} \theta_1x_1^1+\theta_2x_2^1 +...+\theta_nx_n^1\\ \theta_1x_1^2+\theta_2x_2 ^2+...+\theta_nx_n^2\\ \theta_1x_1^3+\theta_2x_2 ^3+...+\theta_nx_n^3\\ \quad... . \\ \quad... .\\ \theta_1x_1^m+\theta_2x_2^m +...+\theta_nx_n^m\\ \end{bmatrix}=X \cdot \theta\tag 9$

$\$
$\$

• 最后进行变换
  $\$

$岭回归代价函数\Rightarrow J(\theta_1,\theta_2...\theta_n)= \frac{1}{m}\left\{\sum_{i=1}^m (h_\theta(x^i)-y^i)^2+ \lambda\sum_{j=1}^n\theta^2\right\}\tag 2$

$\$

$\Rightarrow J(\theta_1,\theta_2...\theta_n)= \frac{1}{m} \left\{ \left[ h_\theta(x^1)-y^1\right]^2 + \left[h_\theta(x^2)-y^2\right]^2+ \left[ h_\theta(x^3)-y^3\right]^2...+ \left[ h_\theta(x^m)-y^m\right]^2 +\lambda(\theta_1^2+\theta_2^2+\theta_3^2+..+\theta_n^2) \right \}$

$套公式\begin{bmatrix}a\\b\\c\end{bmatrix}^T \begin{bmatrix}a\\b\\v\end{bmatrix} =a^2+b^2+c^2$
$所以上上式可改为$
$\Rightarrow J(\theta)=\frac{1}{m} \left\{ \begin{bmatrix} h_\theta(x^1)-y^1\\ h_\theta(x^2)-y^2\\ h_\theta(x^3-y^3\\ ...\\ h_\theta(x^m)-y^m\\ \end{bmatrix}^T\begin{bmatrix} h_\theta(x^1)-y^1\\ h_\theta(x^2)-y^2\\ h_\theta(x^3-y^3\\ ...\\ h_\theta(x^m)-y^m\\ \end{bmatrix}+\lambda\theta^T\theta \right\}$

$\$

$\Rightarrow J(\theta)=\frac{1}{m} \left\{ \left(\begin{bmatrix} h_\theta(x^1)\\ h_\theta(x^2)\\ h_\theta(x^3\\ ...\\ h_\theta(x^m)\\ \end{bmatrix}-\begin{bmatrix} y^1\\y^2\\y^3\\...\\y^m\\ \end{bmatrix} \right)^T\left( \begin{bmatrix} h_\theta(x^1)\\ h_\theta(x^2)\\ h_\theta(x^3\\ ...\\h_\theta(x^m)\\ \end{bmatrix}-\begin{bmatrix} y^1\\-y^2\\y^3\\...\\y^m\\ \end{bmatrix} \right)+\lambda\theta^T\theta \right\}$
$\$
$将子式带入$

$J(\theta)=\frac{1}{m} \left[ (X\theta-Y)^T(X\theta-Y)+\lambda\theta^T\theta \right]$

$\Rightarrow J(\theta)= \frac{1}{m} [(\theta^TX^TX\theta-\theta^TX^TY -Y^TX\theta+Y^TY+\lambda\theta^T\theta]$
$\$
$查表求导得 \\\frac{dJ(\theta)}{d\theta}=\frac{1}{m}[2X^TX\theta -2X^TY+\lambda\theta]$
$\$
$令上式等于0得(此处的\lambda经过减半变换) \\ \theta=(X^TX-\lambda I)^{-1}X^TY$