线性回归分析——含python代码

假设样本空间为 $d$ 个维度,用 $\boldsymbol{x}$ ={ $x_1,x_2,x_3,...,x_d$ }来表示一个样本点，线性回归的目标是用 $d+1$ 个参数来拟合样本点的输入和输出。通常我们会将 $\boldsymbol{x}$ 扩充为 $d+1$ 维的向量 $\boldsymbol{x}$ ={ $x_0,x_1,x_2,x_3,...,x_d$ }，第 $x_0$ 设为1作为偏置项。线性回归表达式如下：

f = \sum_{i = 0}^{d + 1} θ_{i} x_{i}

$f=\sum_{i=0}^{d+1} \theta_i x_i$
用向量可以表示为：

f (x) = θ^{T} x

$f(x)=\boldsymbol{\theta}^T \boldsymbol{x}$
将表达式代入均方误差函数

(M S E)

$(MSE)$ 中:

J = \frac{1}{N} \sum_{i = 1}^{N} (y_{i} - f (x_{i}))^{2}

$J = \frac{1}{N}\sum_{i=1}^{N}(y_i-f(\boldsymbol{x}_i))^2$
令全部样本表示为：

X

$X$ ={

x_{1}, x_{2}, x_{3}, . . ., x_{N}

$\boldsymbol{x}_1,\boldsymbol{x}_2,\boldsymbol{x}_3,...,\boldsymbol{x}_N$ } ^$T$，对应输出表示为

Y

$Y$ ={

y_{1}, y_{2}, y_{3}, . . ., y_{N}

$y_1,y_2,y_3,...,y_N$ } ^$T$，损失函数可以简化为：

J = \frac{1}{N} | | X θ - Y | |^{2}

$J =\frac{1}{N}||X\boldsymbol{\theta}-Y||^2$
我们的目标是让损失函数最小，令其对

θ

$\boldsymbol{\theta}$ 的偏导数为0,则有：

\begin{array}{rcl} (1) & \frac{\partial J}{\partial θ} & = & \frac{\partial}{\partial θ} [\frac{1}{N} (θ^{T} X^{T} X θ - 2 θ^{T} X^{T} Y + Y^{T} Y)] \\ (2) & = & \frac{1}{N} (2 X^{T} X θ - 2 X^{T} Y) \\ (3) & = & 0 \end{array}

$\begin{eqnarray} \frac{\partial J}{\partial \boldsymbol{\theta}}&=&\frac{\partial}{\partial \boldsymbol{\theta}}[\frac{1}{N}(\boldsymbol{\theta}^TX^TX\boldsymbol{\theta}-2\boldsymbol{\theta}^TX^TY+Y^TY)] \\ &=&\frac{1}{N}(2X^TX\boldsymbol{\theta}-2X^TY)\\ &=&0 \end{eqnarray}$
解得：

θ = (X^{T} X)^{- 1} X^{T} Y

$\boldsymbol{\theta}=(X^TX)^{-1}X^TY$ 。
显然，

X^{T} X

$X^TX$ 要是可逆的，通常情况下都能满足因为

N >> d + 1

$N>>d+1$ 。事实上

X^{T} X

$X^TX$ 也存在不可逆的情况，这种情况下我们可以选择求伪逆或者梯度下降法来解决问题。
如果要利用线性回归来解决二分类问题，

y_{i}

$y_i$ 就不再是连续值，而是1(正样本)或者0/-1(负样本)。同样的求出权重向量后对测试样本进行预测，可以用0.5/0作为阈值来划分正负样本。

代码块

理解了线性回归和梯度下降的基本原理，用python撸出来也就10分钟的时间：

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import scale
from random import random
from sklearn.model_selection import train_test_split

class LinearRegression(object):
    weight = np.array([])
    way = 'gd'
    def __init__(self, training_way = 'gd'):
        self.way = training_way
    def gradientDescent(self, X, Y, alpha, epoch):
        W = np.random.normal(0,1,size=(X.shape[1],))
        for i in range(epoch):
            W -= alpha*(X.T).dot(X.dot(W)-Y)/X.shape[0]
        return W

    def fit(self, train_data, train_target, alpha = 0.1, epoch = 300):
        X = np.ones((train_data.shape[0], train_data.shape[1]+1))
        X[:,0:-1] = train_data
        Y = train_target
        if self.way == 'gd':
            self.weight = self.gradientDescent(X, Y, alpha, epoch)
        else:
            self.weight = np.linalg.inv((X.T).dot(X)).dot(X.T).dot(Y)

    def predict(self, test_data):
        X = np.ones((test_data.shape[0], test_data.shape[1]+1))
        X[:,0:-1] = test_data
        return X.dot(self.weight)

    def evaluate(self, predict_target, test_target):
        predict_target[predict_target>=0.5] = 1
        predict_target[predict_target<0.5] = 0
        return sum(predict_target==test_target)/len(predict_target)

if __name__ == "__main__":
    cancer = load_breast_cancer()
    xtr, xval, ytr, yval = train_test_split(cancer.data, cancer.target, \
    test_size=0.2, random_state=7)
    linear = LinearRegression(training_way = 'gd')
    linear.fit(xtr, ytr, alpha = 0.05, epoch = 1000)
    predict = linear.predict(xval)
    print('linear regression accruacy:',linear.evaluate(predict, yval))

线性回归分析——含python代码

代码块

猜你喜欢