Python programming: gradient descent algorithm using multiple linear regression equation, and solving the least squares method with the accuracy of comparison

Three gradient descent algorithm for solving multiple linear regression equation:

(1) questions such as:

To obtain the monthly turnover of goods of a shop area with large shop area of ​​relevance, or associated with the store large distance from the station, we need to shop area, the distance from the station, as well as the establishment of a monthly turnover linear regression equation and solve the equation, and correlation coefficient:

The table data entry excel spreadsheet, save the file as mytest.csv:

The first column is the shop area, the nearest station from the second column, third column on turnover.

(2) Python Programming:

1. The complete code

import numpy as np
from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
data=np.genfromtxt('mytest.csv',delimiter=',')
x_data=data[:,:-1]
y_data=data[:,2]
#定义学习率、斜率、截据
#设方程为y=theta1x1+theta2x2+theta0
lr=0.00001
theta0=0
theta1=0
theta2=0
#定义最大迭代次数，因为梯度下降法是在不断迭代更新k与b
epochs=10000
#定义最小二乘法函数-损失函数（代价函数）
def compute_error(theta0,theta1,theta2,x_data,y_data):
    totalerror=0
    for i in range(0,len(x_data)):#定义一共有多少样本点
        totalerror=totalerror+(y_data[i]-(theta1*x_data[i,0]+theta2*x_data[i,1]+theta0))**2
    return totalerror/float(len(x_data))/2
#梯度下降算法求解参数
def gradient_descent_runner(x_data,y_data,theta0,theta1,theta2,lr,epochs):
    m=len(x_data)
    for i in range(epochs):
        theta0_grad=0
        theta1_grad=0
        theta2_grad=0
        for j in range(0,m):
            theta0_grad-=(1/m)*(-(theta1*x_data[j,0]+theta2*x_data[j,1]+theta2)+y_data[j])
            theta1_grad-=(1/m)*x_data[j,0]*(-(theta1*x_data[j,0]+theta2*x_data[j,1]+theta0)+y_data[j])
            theta2_grad-=(1/m)*x_data[j,1]*(-(theta1*x_data[j,0]+theta2*x_data[j,1]+theta0)+y_data[j])
        theta0=theta0-lr*theta0_grad
        theta1=theta1-lr*theta1_grad
        theta2=theta2-lr*theta2_grad
    return theta0,theta1,theta2
#进行迭代求解
theta0,theta1,theta2=gradient_descent_runner(x_data,y_data,theta0,theta1,theta2,lr,epochs)
print('迭代次数：{0} 学习率：{1}\na0={2}\na1={3}\na2={4}'.format(epochs,lr,theta0,theta1,theta2))
print("多元线性回归方程为:y=",theta1,"X1+",theta2,"X2+",theta0)
#画图
ax=plt.figure().add_subplot(111,projection='3d')
ax.scatter(x_data[:,0],x_data[:,1],y_data,c='r',marker='o')
x0=x_data[:,0]
x1=x_data[:,1]
#生成网格矩阵
x0,x1=np.meshgrid(x0,x1)
z=theta0+theta1*x0+theta2*x1
#画3d图
ax.plot_surface(x0,x1,z)
ax.set_xlabel('area')
ax.set_ylabel('distance')
ax.set_zlabel("Monthly turnover")
plt.show()

2. Run results

Contrast on the comic formula

Found errors.

Fourth, using the least squares method to solve multiple linear regression equation:

(1) python programming

1. The complete code

#利用线性代数的矩阵模拟最小二乘法求解法求解多元线性回归方程的系数
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
%matplotlib inline
data = np.genfromtxt("mytest.csv",delimiter=",")
X1=data[0:10,0]#自变量温度
X2=data[0:10,1]#因变量销售量
Y=data[0:10,2]#自变量温度
#将因变量赋值给矩阵Y1
Y1=np.array([Y]).T
#为自变量系数矩阵X赋值
X11=np.array([X1]).T
X22=np.array([X2]).T
A=np.array([[1],[1],[1],[1],[1],[1],[1],[1],[1],[1]])#创建系数矩阵
B=np.hstack((A,X11))#将矩阵a与矩阵X11合并为矩阵b
X=np.hstack((B,X22))#将矩阵b与矩阵X22合并为矩阵X
#求矩阵X的转置矩阵
X_=X.T
#求矩阵X与他的转置矩阵的X_的乘积
X_X=np.dot(X_,X)
#求矩阵X与他的转置矩阵的X_的乘积的逆矩阵
X_X_=np.linalg.inv(X_X)
#求解系数矩阵W，分别对应截距b、a1、和a2
W=np.dot(np.dot((X_X_),(X_)),Y1)
b=W[0][0]
a1=W[1][0]
a2=W[2][0]
print("系数a1=",a1)
print("系数a2=",a2)
print("截距为=",b)
print("多元线性回归方程为:y={0}*X1+{1}*X2+{2}".format(a1,a2,b))
#画出线性回归分析图
data1=pd.read_excel('mytest.xlsx')
sns.pairplot(data1, x_vars=['area','distance'], y_vars='sales', height=3, aspect=0.8, kind='reg')  
plt.show() 
#求月销售量Y的和以及平均值y1
sumy=0#因变量的和
y1=0#因变量的平均值
for i in range(0,len(Y)):
    sumy=sumy+Y[i]
y1=sumy/len(Y)
#求月销售额y-他的平均值的和
y_y1=0#y-y1的值的和
for i in range(0,len(Y)):
    y_y1=y_y1+(Y[i]-y1)
print("销售量-销售量平均值的和为:",y_y1)
#求预测值sales1
sales1=[]
for i in range(0,len(Y)):
    sales1.append(a1*X1[i]+a2*X2[i]+b)
#求预测值的平均值y2
y2=0
sumy2=0
for i in range(len(sales1)):
    sumy2=sumy2+sales1[i]
y2=sumy2/len(sales1)
#求预测值-平均值的和y11_y2
y11_y2=0
for i in range(0,len(sales1)):
   y11_y2=y11_y2+(sales1[i]-y2)
print("预测销售值-预测销售平均值的和为:",y11_y2)
#求月销售额y-他的平均值的平方和
Syy=0#y-y1的值的平方和
for i in range(0,len(Y)):
    Syy=Syy+((Y[i]-y1)*(Y[i]-y1))
print("Syy=",Syy)
#求y1-y1平均的平方和
Sy1y1=0
for i in range(0,len(sales1)):
    Sy1y1=Sy1y1+((sales1[i]-y2)*(sales1[i]-y2))
print("Sy1y1=",Sy1y1)
#（y1-y1平均）*（y-y平均）
Syy1=0
for i in range(0,len(sales1)):
    Syy1=Syy1+((Y[i]-y1)*(sales1[i]-y2))
print("Syy1=",Syy1)
#求R
R=Syy1/((Syy*Sy1y1)**0.5)
R2=R*R
print("判定系数R2=",R2)

2. Run results

Results (2) Comparative least-squares method and gradient descent algorithm

Comparison three results

Gradient descent algorithm:

The least squares method:

Comic results:

By comparison of the results of both found that the progress of the least square method than gradient descent algorithm. Gradient descent of several iterations to complete, but progress is not high, it is more waste, basically abandoned that method.

V. summary

Compared gradient descent method and the least squares method, gradient descent method step size to choose, but does not require the least squares method. Gradient descent is an iterative solution method is a least squares calculation of analytical solution. If the sample size is not large, and there analytical solution, least squares method have advantages compared to gradient descent method to calculate fast. However, if the sample size is large, due to the request by the least squares method a super inverse matrix, then it is difficult or slow to solve the analytical solution, and has an advantage of using an iterative gradient descent method.