Implementation of handwritten gradient descent for linear regression of y=kx+b

Algorithm steps:

(1) Construct data, y=3*x+5;

(2) Random initialization $\hat{k}$ sum $\hat{b}$ , any value, such as $\hat{k}$ =9, $\hat{b}$ =10;

(3) Calculate $\hat{y}$ , $\hat{y}=\hat{k}x+\hat{b}$ , and calculate $loss=(y-\hat{y})^{^{2}}=(y-(\hat{k}x+\hat{b}))^{2}$

(4) Calculate the derivatives of $\hat{k}$ and respectively , , $\hat{b}$ $\hat{k}=\hat{k}-\Delta \hat{k}$ $\hat{b}=\hat{b}-\Delta \hat{b}$

$\Delta \hat{k}={\frac{\partial loss}{\partial \hat{k}}}=-2(y-({\hat{k}x+\hat{b})})\cdot x$

$\Delta \hat{b}={\frac{\partial loss}{\partial \hat{b}}}=-2(y-({\hat{k}x+\hat{b})})$

Stop after repeating the loop n times

Construct a linear function:

$y=kx+b\ \ (k=3,b=5)$

Code:

X=[i for i in range(0,15)]

k=3
b=5
Y=[k*i+b for i in X]

import matplotlib.pyplot as plt
fig, ax = plt.subplots()

# 在同一个图形中绘制散点图和折线图
ax.scatter(X, Y, color='blue', label='scatter')
ax.plot(X, Y, color='red', label='line')

# 添加图例
ax.legend()

# 显示图形
plt.show()

MSE loss function:

$loss=(y-\hat{y})^{^{2}}=(y-(\hat{k}x+\hat{b}))^{2}$

loss.append((Y[i]-y_[i])**2)  #公式对应代码

The results of deriving the derivatives of k and b respectively are as shown in the figure:

${\frac{\partial loss}{\partial \hat{k}}}=-2(y-({\hat{k}x+\hat{b})})\cdot x$

${\frac{\partial loss}{\partial \hat{b}}}=-2(y-({\hat{k}x+\hat{b})})$

#公式对应代码
delta_K_sum.append((Y[i]-y_[i])*(-2)*X[i])
delta_B_sum.append((Y[i]-y_[i])*(-2))

All code:

X=[i for i in range(0,15)]
X
k=3
b=5
Y=[k*i+b for i in X]
Y
import matplotlib.pyplot as plt
# 创建散点图
plt.scatter(X, Y)

# 显示图形
plt.show()
#随机初始化要求的k和b
K=8
B=10
#k和b是正确答案，根据数据和随机初始化的K和B去拟合函数，找到最优的k和b
#y=Kx+B
loss=[]

#计算预测值
for i in range(1000):
    y_=[K*i+B for i in X]

    loss=[]
    for i in range(len(X)):
        loss.append((Y[i]-y_[i])**2)
    print(sum(loss)/len(loss))
    # cha=loss.sum()/len(loss)
    #计算loss

    #根据最小二乘法  对y_求导，等我用纸写一下，利用loss对K求梯度，去更新K的值，对B求梯度，求更新B的值
    #直到K和B基本拟合图像
    delta_K_sum=[]
    delta_B_sum=[]
    for i in range(len(X)):
        delta_K_sum.append((Y[i]-y_[i])*(-2)*X[i])
        delta_B_sum.append((Y[i]-y_[i])*(-2))
    delta_K=sum(delta_K_sum)/len(delta_K_sum)

    delta_B=sum(delta_B_sum)/len(delta_B_sum)
    #0.01是学习率，保证稳定收敛
    K=K-0.01*delta_K
    B=B-0.01*delta_B
    print(K,B)
print(K,B)

Result image:

X=[i for i in range(0,15)]

Y=[K*i+B for i in X]

import matplotlib.pyplot as plt
fig, ax = plt.subplots()

# 在同一个图形中绘制散点图和折线图
ax.scatter(X, Y, color='blue', label='scatter')
ax.plot(X, Y, color='red', label='line')

# 添加图例
ax.legend()

# 显示图形
plt.show()

[Machine Learning] Python basics to implement linear regression

Algorithm steps:

Construct a linear function:

MSE loss function:

Result image:

Guess you like