Python 算法：线性回归及相关公式推导

0 前言

本文以一个小小的案例展开，主要讲解了线性回归的步骤、常用的两种求最优解的方法（最小二乘法和sklearn回归算法及算法原理）及相关函数、公式的过程推导。

相关环境：
Windows 64位
Python3.9
scikit-learn==1.0.2
pandas==1.4.2
numpy==1.21.5
matplotlib==3.5.1

1 案例数据及数据关系

假设有一组数如下，问(10,27)是否合理？

x	y
1	5
2	6
3	9
4	11
5	13

要回答这个问题，可以分三步走：
1、确认x和y的关系；
2、拟合模型，并根据模型进行预测；
3、判断(10,27)是否合理。

要确定关系，可以将数据通过散点图绘制出来

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame({
    
    'x':list(range(1,6)),'y':[5,6,9,11,13]})
plt.scatter(df.x,df.y)
plt.show()

由散点图可见，这是一个线性回归模型，故令线性模型函数为$f\left(x_i\right)=a{x}_i+b$，使得$f\left(x_i\right)\approx y_i$。
这时，一开始的问题就转化成了求系数a和截距b。

2 拟合模型，求最优解

When do a and b reach the optimal solution, that is, the best fitting effect?
According to the past mathematical experience, it is to draw a straight line and pass as many scattered points as possible or close to the scattered points, but it is not sure whether this line is optimal. combined with the results to determine. To judge whether the fitting result is good or bad, there is a common judgment method: the value obtained by the least square method (LS) is as small as possible.
In layman's terms, the least squares method is to take the actual value of each point $y_i$And the predicted value of the line $f\left(x_i\right)$ are subtracted, then squared, then summed.
In special cases, when the result value is 0, it means that the actual point and the point of the straight line are completely coincident, and they are actually the points taken out from the straight line.
The formula of the least square method is as follows:

$LS={\sum }_{i=1}^m{\left(f\left(x_i\right)-y_i\right)}^2,\; wheref\left(x_i\right)=a{x}_i+b\text{(2.1)}$

均分误差（MSE，也称平方损失）的公式如下：

$MSE=\frac{1}{m}{\sum }_{i=1}^m{\left(f\left(x_i\right)-y_i\right)}^2，其中f\left(x_i\right)=a{x}_i+b\text{(2.2)}$
注：m 表示点的总数，i 表示每个点。

二者差别在于最小二乘法没有除以m，均方差除以m。在某些场景会有变形，比如加权最小二乘法和加权均方差，权重是对应值出现的概率。
在周志华的机器学习一书讲到的一个概念是“最小二乘法是基于均方误差最小化来进行模型求解的方法”，即：
$\boxed{minimizeMSE\stackrel{模型}{\Rightarrow }LS}$
介绍完LS和MSE，再回到一开始的问题，什么时候a和b达到最优解，可以通过求导方式求解，即在均方差函数上分别对相关未知变量求导，再令导函数$f^{\prime }=0$进行求解；也可以通过算法进行定向的搜索求解，即先分别给a和b先赋一个初始值，然后通过梯度下降算法，不断更新a和b的值，最终收敛到极值，得到最优解。

2.1 均方差求导求解

如何在均方差$E\left(a,b\right)=\frac{1}{m}{\sum }_{i=1}^m{\left(f\left(x_i\right)-y_i\right)}^2$上，分别对a和b求导呢？
先把$f\left(x_i\right)=a{x}_i+b$代入上式，得到

$\begin{array}{rlr}E\left(a,b\right)& =\frac{1}{m}\sum _{i=1}^m{\left(a{x}_i+b-y_i\right)}^2& (2.3)\\ & =\frac{1}{m}\sum _{i=1}^m\left(x_i^2{a}^2+2\left(b-y_i\right)x_{i}a+{\left(b-y_i\right)}^2\right)& (2.4)\\ & =\frac{1}{m}\sum _{i=1}^m\left(b^2+2\left(a{x}_i-y_i\right)b+{\left(a{x}_i-y_i\right)}^2\right)& (2.5)\end{array}$

When deriving a, it can be changed to the structure of formula (2.4) for viewing; when deriving b, it can be changed to the structure of formula (2.5) for viewing. The result of derivation is:

$\begin{array}{rlr}\frac{\partial }{\partial a}{E}_{(a,b)}& =\frac{2}{m}\sum _{i=1}^m\left(a{x}_i+b-y_i\right)x_i& (2.6)\\ \frac{\partial }{\partial b}{E}_{(a,b)}& =\frac{2}{m}\sum _{i=1}^m\left(a{x}_i+b-y_i\right)& (2.7)\end{array}$

Let (2.6) and (2.7) both be 0, and combine the two newly obtained equations to get the solution:

$a=\frac{\sum _{i=i}^m\left(y_1-\overline{y}\right)x_i}{\sum _{i=i}^m\left(x_1-\overline{x}\right)x_i}or\frac{\sum _{i=1}^m\left(x_i-\overline{x}\right)y_i}{\sum _{i=i}^m\left(x_1-\overline{x}\right)x_i}or\frac{\sum _{i=i}^m\left(y_1-\overline{y}\right)\left(x_i-\overline{x}\right)}{\sum _{i=i}^m{\left(x_1-\overline{x}\right)}^2}or=\frac{\sum _{i=i}^m{x}_i{y}_1-m\overline{x}\overline{y}}{\sum _{i=i}^m{x}_i^2-m{\overline{x}}^2}\text{(2.9)}$


$b=\frac{1}{m}\sum _{i=1}^m\left(y_i-a{x}_i\right)=\overline{y}-a\overline{x}\text{(2.10)}$

Note: The 4 numerators of a get the same value, as do the 3 denominators, and there can be many combinations. The specific derivation process of the above formula can be found in the formula derivation section.

The solution obtained from this set of data is a=2.1; b=2.5.

import pandas as pd
import matplotlib.pyplot as plt
df = pd.DataFrame({
    
    'x':list(range(1,6)),'y':[5,6,9,11,13]})
plt.figure(figsize=(8,6))
plt.scatter(df.x,df.y)
plt.plot(df.x,df.x*2.1+2.5)
plt.show()

print(2.1*10+2.5)           # 结果为23.5

The fitting results are as follows:

Interlude: The Role of Derivatives

The role of the derivative: The derivative describes the rate of change of a function at a certain point, and the magnitude of this rate of change needs to be defined according to the result of the derivative.
For example, a straight line function (such as $y=x$ , its derivative function is$y^{\prime }=1$ ) the derivative is a natural number ($Number1$ ), then the derivative of any point on the whole line is this value, and the rate of change remains unchanged.
If it is a curve function (such as$y=x^2$ , its derivative function is$y^{\prime }=2x$ ), its derivative will give a rate of change ($y^{\prime }$ ), also expressed as the slope of the tangent line at the point, the smaller the slope of the tangent line, the closer to the global or local maximum value (maximum or minimum value), when the tangent line slope is 0, it reaches the maximum value (may be global or possible is local, combined with function judgment, the highest power is 2 times, then there is only one maximum value), as shown in Figure 1 below.
The role of the derivative is to help us find the maximum value when the slope of the tangent line is equal to 0. Generally, this value is the optimal solution of the original function. Of course, if a function has multiple tangent slopes equal to 0, a local optimal solution may be obtained instead of a global optimal solution. (as shown in Figure 2 below)

2.2 Algorithm fitting solution

Next, let's look at the way the algorithm is implemented. The process of algorithm implementation is through directional search.
First look at a picture, as shown in the figure below, which is also a search process, directly looking for a line with a better fitting effect on the original scattered points. However, it is difficult for this method to judge when the optimal solution can be reached.

At this time, a more scientific way is needed to solve the above problems. The predecessors have created many algorithms for us to search for the optimal solution. One of these algorithms is the classic gradient descent method . This algorithm has some similarities with the previous derivative function. When looking for the extremum, the gradient corresponding to the function is continuously iterated and searched, and finally converges at the extremum point. As shown in the figure below:

This is a two-dimensional display. If it is a three-dimensional display, you can refer to a classic picture of Wu Enda's machine learning course:
Note: $J\left(i_0,i_1\right)$ is the cost function, same as J ( a , b ) J \left( a,b \right)in this paper$J\left(a,b\right)$ ,$i_0$Sum $i_1$Then analogy a and b is the optimal solution of the cost function (discussed below).

The algorithm logic of gradient descent is very simple, but it will be a little complicated to express it with formulas and solutions.

Before talking about the algorithm, introduce another concept: loss function (also called cost function ). As the algorithm continues to evolve, so does the loss function. This article builds a more classic one, which is based on the evolution of the mean square error, that is, the calculation result of the mean square error is divided by 2. After reading a lot of information, the introduction of this 2 is mainly used to reduce the coefficient 2 after the mean square error is derived. , for the convenience of calculation, let’s understand it this way.
The loss function introduced this time is as follows:

$J\left(a,b\right)=\frac{1}{2m}{\sum }_{i=1}^m{\left(f\left(x_i\right)-y_i\right)}^2\text{(2.11)}$

Note: The difference between the loss function and the cost function is that the loss function is defined in a single sample, which is the error of a sample, and the cost function is defined in the entire training set, which is the average error of all samples.

Let us generate a continuous equation:

$\begin{array}{rlr}{a}_{i+1}& =a_i-a\frac{\partial }{\partial a_i}{J}_{(a,b)}& (2.12)\\ b_{i+1}& =b_i-a\frac{\partial }{\partial b_i}{J}_{(a,b)}& (2.13)\end{array}$

These two formulas are iterative formulas, which are the results obtained by subtracting the following long list of formulas from the a and b values ​​obtained this time for the next a and b values.
$\alpha$ is the learning rate, or step size, which is the value you define for the algorithm, how much you expect to move each time;$\frac{\partial }{\partial a}{J}_{(a,b)}$和$\frac{\partial }{\partial b}{J}_{(a,b)}$是求导公式，即在损失函数上，对a和b分别求导。
补充说明：可能有的小伙伴对于对a和b分别求导有点无厘头，特别是多个位置参数放在一起的时候，更容易混淆。无论如何复杂，认准一点，对谁求导，谁就是自变量，其他的全部视为常数，然后根据求导的规则获取导函数。比如：$f\left(x\right)=a{x}^2+bx+c$对a求导就是$f^{\prime }\left(a\right)=x^2$，对b求导就是$f^{\prime }\left(b\right)=x$，对c求导就是$f^{\prime }\left(c\right)=1$。

如何在代价函数$J\left(a,b\right)=\frac{1}{2m}{\sum }_{i=1}^m{\left(f\left(x_i\right)-y_i\right)}^2$上，分别对a和b求导呢？这个和上文讲均方差求导时有介绍过类似内容，只不过前面是均方差，此处是代价函数，多除以2，其他基本一致。
先把$f\left(x_i\right)=a{x}_i+b$代入上式，得到

$\begin{array}{rlr}J\left(a,b\right)& =\frac{1}{2m}\sum _{i=1}^m{\left(a{x}_i+b-y_i\right)}^2& (2.14)\\ & =\frac{1}{2m}\sum _{i=1}^m\left(x_i^2{a}^2+2\left(b-y_i\right)x_{i}a+{\left(b-y_i\right)}^2\right)& (2.15)\\ & =\frac{1}{2m}\sum _{i=1}^m\left(b^2+2\left(a{x}_i-y_i\right)b+{\left(a{x}_i-y_i\right)}^2\right)& (2.16)\end{array}$

When deriving a, it can be changed to the structure of formula (2.15) for viewing; when deriving b, it can be changed to the structure of formula (2.16) for viewing. The result of derivation is:

$\begin{array}{rlr}\frac{\partial }{\partial a}{J}_{(a,b)}& =\frac{1}{m}\sum _{i=1}^m\left(a{x}_i+b-y_i\right)x_i& (2.17)\\ \frac{\partial }{\partial b}{J}_{(a,b)}& =\frac{1}{m}\sum _{i=1}^m\left(a{x}_i+b-y_i\right)& (2.18)\end{array}$

Note: For the specific derivation process, please refer to the formula derivation section.

将(2.17)代入(2.12)，将(2.18)代入(2.13)，便可以得到：

$\begin{array}{rlr}{a}_{i+1}& =a_i-a\frac{1}{m}\sum _{i=1}^m\left(a_i{x}_i+b_i-y_i\right)x_i& (2.19)\\ b_{i+1}& =b_i-a\frac{1}{m}\sum _{i=1}^m\left(a_i{x}_i+b_i-y_i\right)& (2.20)\end{array}$

注：右边的a和b的结构都是本次的a和b，左边的a和b是下一次值，二者同步更新。用Python实现过程，或需要新建一个中间变量，以避免在计算完(2.19)式之后，变量a 变成了下一次的值，等到计算(2.20)式时，使用下一次的a 值更新下一次的b 值。

如果通过Python实现们可以直接调用scikit-learn（sklearn）的线性回归模型库实现，具体代码实现步骤如下：

# 读取数据
import pandas as pd
data = pd.DataFrame({
    
    'x':[1,2,3,4,5],'y':[5,6,9,11,13]})
# data.shape,type(data)
x = data.loc[:,'x']
y = data.loc[:,'y']

# 画图查看数据分布
from matplotlib import pyplot as plt
plt.figure(figsize=(8,6))
plt.scatter(x,y)
plt.show()

# 调用sklearn的线性模型
from sklearn.linear_model import LinearRegression
lr_model = LinearRegression()

# 转换维度，一维转为二维
X = x.values.reshape(-1, 1) 

# 训练模型
lr_model.fit(X,y)
# 预测x结果
y_predict = lr_model.predict(X)

# 画图查看y和y_predict
plt.figure(figsize=(8,6))
plt.scatter(x,y)
plt.plot(x,y_predict)
plt.show()

# 打印a、b值
a = lr_model.coef_
b = lr_model.intercept_
print(a,b)                  # 结果为：[2.1] 2.499999999999999

# 查看均方差
from sklearn.metrics import mean_squared_error,r2_score
MSE = mean_squared_error(y,y_predict)
R2 = r2_score(y,y_predict)
print(MSE,R2)               # 结果为：0.13999999999999996 0.984375

# 预测具体某个值的结果
y_p = lr_model.predict([[10]])
print(y_p)                  # 结果为：[23.5]

拟合的曲线为：$f\left(x_i\right)=2.1x_i+2.5$
拟合结果可视化如下：

3 判断结果

通过以上两种求解方法得到的结果均为23.5。(10,27)点在拟合直线的上方，27比23.5大3.5。
是否合理呢？可以通过$R^2$来衡量，在算法计算中有计算过该值，为0.984375，也就是说，在该模型的拟合效果上，有98.4%的可信度相信当x=10时，得到23.5。所以27是不合理的。但是，需要注意的是，该可信度是基于已知的5个点训练得到的结果，由于数量较少，对x>5之外的值可能不适用，当然，如果是相同的场景下，或有一定的适用性，权当教学使用。
虽然27不合理，但是在涉及到决策时，需要再结合实际的情况做出是否接受的判断，比如说如果是某地的房价预测是23.5万/平，实际出价是27万/平，对于卖方有利，对于卖方无利。站在卖方角度，一般可接受，站在买方角度，一般不可接受。

4 Formula derivation

4.1 如何证明${\sum }_{i=1}^m\left(x_i-\overline{x}\right)x_i={\sum }_{i=1}^m{\left(x_i-\overline{x}\right)}^2={\sum }_{i=i}^m{x}_i^2-m{\overline{x}}^2$ established?

$\sum _{i=1}^m{x}_i=x_1+x_2+x_3+\cdot \cdot \cdot \cdot \cdot \cdot +x_m=m\overline{x}=\sum _{i=1}^m\overline{x}\text{(4.1)}$

In formula (4.1), the sum of m sample values ​​​​of x is equal to m times the mean value, which can be expressed as${\sum }_{i=1}^m\overline{x}$, ie m $\overline{x}$相加。

$\sum _{i=1}^m\left(\overline{x}{x}_i\right)=\overline{x}\sum _{i=1}^m{x}_i\text{(4.2)}$

(4.2),$\overline{x}$is a constant, the formula (4.2) is equivalent to $\overline{x}{x}_1+\overline{x}{x}_2+\cdot \cdot \cdot +\overline{x}{x}_m=\overline{x}(x_1+x_2+\cdot \cdot \cdot +x_m)$

证明${\sum }_{i=1}^m\left(x_i-\overline{x}\right)x_i={\sum }_{i=1}^m{\left(x_i-\overline{x}\right)}^2$

$\begin{array}{rlr}\sum _{i=1}^m\left(x_i-\overline{x}\right)x_i& =\sum _{i=1}^m\left(x_i^2-\overline{x}{x}_i\right)& (4.3)\\ & =\sum _{i=1}^m\left(x_i^2-\overline{x}{x}_i+\overline{x}{x}_i-\overline{x}{x}_i\right)& (4.4)\\ & =\sum _{i=1}^m\left(x_i^2-2\overline{x}{x}_i+\overline{x}{x}_i\right)& (4.5)\\ & =\sum _{i=1}^m\left(x_i^2-2\overline{x}{x}_i\right)+\sum _{i=1}^m\left(\overline{x}{x}_i\right)& (4.6)\\ & =\sum _{i=1}^m\left(x_i^2-2\overline{x}{x}_i\right)+\overline{x}\sum _{i=1}^m\left(x_i\right)& (4.7)\\ & =\sum _{i=1}^m\left(x_i^2-2\overline{x}{x}_i\right)+\overline{x}\cdot m\overline{x}& (4.8)\\ & =\sum _{i=1}^m\left(x_i^2-2\overline{x}{x}_i\right)+m{\overline{x}}^2& (4.9)\\ & =\sum _{i=1}^m\left(x_i^2-2\overline{x}{x}_i\right)+\sum _{i=1}^m{\overline{x}}^2& (4.10)\\ & =\sum _{i=1}^m\left(x_i^2-2\overline{x}{x}_i+{\overline{x}}^2\right)& (4.11)\\ & =\sum _{i=1}^m{\left(x_i-\overline{x}\right)}^2& (4.12)\end{array}$

证明${\sum }_{i=1}^m\left(x_i-\overline{x}\right)x_i={\sum }_{i=i}^m{x}_i^2-m{\overline{x}}^2$：

$\begin{array}{rlr}\sum _{i=1}^m\left(x_i-\overline{x}\right)x_i& =\sum _{i=1}^m{x}_i^2-\sum _{i=1}^m\overline{x}{x}_i& (4.13)\\ & =\sum _{i=1}^m{x}_i^2-\overline{x}\sum _{i=1}^m{x}_i& (4.14)\\ & =\sum _{i=1}^m{x}_i^2-\overline{x}\cdot m\overline{x}& (4.15)\\ & =\sum _{i=1}^m{x}_i^2-m{\overline{x}}^2& (4.16)\end{array}$
结合(4.1)式代入(4.14)可得到(4.15)式。

4.2 如何证明${\sum }_{i=1}^m\left(x_i-\overline{x}\right)y_i={\sum }_{i=1}^m\left(y_i-\overline{y}\right)x_i={\sum }_{i=i}^m\left(y_i-\overline{y}\right)\left(x_i-\overline{x}\right)={\sum }_{i=i}^m{x}_i{y}_i-m\overline{x}\overline{y}$成立？


$\begin{array}{rl}\overline{x}=\frac{1}{m}\sum _{i=1}^m{x}_i& (4.17)\\ \overline{y}=\frac{1}{m}\sum _{i=1}^m{y}_i& (4.18)\end{array}$

证明${\sum }_{i=1}^m\left(x_i-\overline{x}\right)y_i={\sum }_{i=1}^m\left(y_i-\overline{y}\right)x_i$：

$\begin{array}{rlr}\sum _{i=1}^m\left(x_i-\overline{x}\right)y_i& =\sum _{i=1}^m\left(x_i{y}_i-\overline{x}{y}_i\right)& (4.19)\\ & =\sum _{i=1}^m{x}_i{y}_i-\sum _{i=1}^m\overline{x}{y}_i& (4.20)\\ & =\sum _{i=1}^m{x}_i{y}_i-\overline{x}\cdot \sum _{i=1}^m{y}_i& (4.21)\\ & =\sum _{i=1}^m{x}_i{y}_i-\left(\frac{1}{m}\sum _{i=1}^m{x}_i\right)\cdot \left(m\overline{y}\right)& (4.22)\\ & =\sum _{i=1}^m{x}_i{y}_i-\overline{y}\sum _{i=1}^m{x}_i& (4.23)\\ & =\sum _{i=1}^m{x}_i{y}_i-\sum _{i=1}^m\overline{y}{x}_i& (4.24)\\ & =\sum _{i=1}^m\left(x_i{y}_i-\overline{y}{x}_i\right)& (4.25)\\ & =\sum _{i=1}^m\left(y_i-\overline{y}\right)x_i& (4.26)\end{array}$

Substituting (4.17) and (4.18) into (4.21) yields (4.22).
Note: ${\sum }_{i=1}^m{x}_i{y}_i/=x_i{\sum }_{i=1}^m{y}_i$because $x_i$is not a constant, ${\sum }_{i=1}^m{x}_i{y}_i$Represents two variables $x_i$sum $y_i$The sum of products is calculated in the order of multiplication and then addition. And $x_i{\sum }_{i=1}^m{y}_i$is the variable $y_i$Multiply the variable $x_i$. fruit $x_i$is a constant equality.

证明${\sum }_{i=1}^m\left(x_i-\overline{x}\right)y_i={\sum }_{i=i}^m\left(y_i-\overline{y}\right)\left(x_i-\overline{x}\right)$：

$\begin{array}{rlr}\sum _{i=i}^m\left(y_1-\overline{y}\right)\left(x_i-\overline{x}\right)& =\sum _{i=i}^m\left(x_i{y}_i-\overline{x}{y}_i-x_i\overline{y}+\overline{xy}\right)& (4.27)\\ & =\sum _{i=i}^m\left(x_i-\overline{x}\right)y_i-\sum _{i=i}^m\left(x_i\overline{y}\right)+\sum _{i=i}^m\left(\overline{xy}\right)& (4.28)\\ & =\sum _{i=i}^m\left(x_i-\overline{x}\right)y_i-\overline{y}\sum _{i=i}^m\left(x_i\right)+\overline{y}\sum _{i=i}^m\left(\overline{x}\right)& (4.29)\\ & =\sum _{i=i}^m\left(x_i-\overline{x}\right)y_i-m\overline{xy}+-m\overline{xy}& (4.30)\\ & =\sum _{i=i}^m\left(x_i-\overline{x}\right)y_i& (4.31)\end{array}$
Substitute (4.1) and (4.17) into (4.29) to get (4.30).

证明${\sum }_{i=1}^m\left(x_i-\overline{x}\right)y_i={\sum }_{i=i}^m{x}_i{y}_i-m\overline{x}\overline{y}$: Combining (4.18) and (4.21), the result can be deduced.

4.3 为什么$J_{(a,b)}=\frac{1}{2m}{\sum }_{i=1}^m{\left(a{x}_i+b-y_i\right)}^2$对a 求导之后是$\frac{1}{m}{\sum }_{i=1}^m\left(a{x}_i+b-y_i\right)x_i$, and after derivation to b is $\frac{1}{m}{\sum }_{i=1}^m\left(a{x}_i+b-y_i\right)$ ?

at$And(to,b)$函数上对a和b求导的解法和该解法相类似，不赘述。
$\begin{array}{rlr}J\left(a,b\right)& =\frac{1}{2m}\sum _{i=1}^m{\left(a{x}_i+b-y_i\right)}^2& (4.32)\\ & =\frac{1}{2m}\sum _{i=1}^m\left(x_i^2{a}^2+2\left(b-y_i\right)x_{i}a+{\left(b-y_i\right)}^2\right)& (4.33)\\ & =\frac{1}{2m}\sum _{i=1}^m\left(b^2+2\left(a{x}_i-y_i\right)b+{\left(a{x}_i-y_i\right)}^2\right)& (4.34)\end{array}$

Use formula (4.33) to derive derivatives for a; use formula (4.34) to derive derivatives for b.

$$\begin{gathered} \begin{array}{rlr}\frac{\partial J_{(a,b)}}{\partial a}& =\frac{1}{2m}\sum _{i=1}^m\left(2x{\_}_i^{2}a+2\left(b-y_i\right)x_i+0\right)& (4.35)\\ & =\frac{1}{2m}\sum _{i=1}^{m}2x{\_}_i\left(x_{i}a+\left(b-y_i\right)\right)& (4.36)\\ & =\frac{1}{m}\sum _{i=1}^m{x}_i\left(x_{i}a+b-y_i\right)& (4.37)\\ & =\frac{1}{m}\sum _{i=1}^m\left(a{x}_i+b-y_i\right)x_i& (4.38)\end{array} \\ \begin{array}{rlr}\frac{\partial J_{(a,b)}}{\partial b}& =\frac{1}{2m}\sum _{i=1}^m\left(2b\_+2\left(a{x}_i-y_i\right)+0\right)& (4.39)\\ & =\frac{1}{2m}\sum _{i=1}^{m}2x{\_}_i\left(x_{i}a+\left(b-y_i\right)\right)& (4.40)\\ & =\frac{1}{m}\sum _{i=1}^m{x}_i\left(x_{i}a+b-y_i\right)& (4.41)\\ & =\frac{1}{m}\sum _{i=1}^m\left(a{x}_i+b-y_i\right)x_i& (4.42)\end{array} \end{gathered}$$

4.4 The formula and deformation derivation of sklearn.metrics.r2_score

Basics:
Total Sum of Squares: $TSS={\sum }_{i=1}^m{\left(y_i-\overline{y}\right)}^2$ (the sum of squares of the difference between the actual value and the mean)
regression sum of squares (Explained Sum of Squares):$ESS={\sum }_{i=1}^m{\left(y^{\prime }-\overline{y}\right)}^2$ (the sum of squares of the difference between the predicted value and the mean)
residual sum of squares (Residual Sum of Squares):$RSS={\sum }_{i=1}^m{\left(y_i-y^{\prime }\right)}^2$ (the sum of squares of the difference between actual and predicted values)
$TSS=ESS+RSS$
均方差（Mean Square Error）：$MSE=\frac{1}{m}{\sum }_{i=1}^m{\left(y_i-y^{\prime }\right)}^2=\frac{1}{m}RSS$
y的方差（Variance）：$wa{r}_y=\frac{1}{m}{\sum }_{i=1}^m{\left(y_i-\overline{y}\right)}^2=\frac{1}{m}TSS$
x的方差（Variance）：$wa{r}_x=\frac{1}{m}{\sum }_{i=1}^m{\left(x_i-\overline{x}\right)}^2$
standard deviation (Standard Deviation):$std=\sqrt{war}$

$\begin{array}{rl}{R}^2& =1-\frac{RSS}{TSS}(the\; ratio\; of\; the\; residual\; sum\; of\; squares\; to\; the\; overall\; sum\; of\; squares)\\ & =1-\frac{MSE}{wa{r}_y}(mean\; square\; error\; and{y}_{i}The\; ratioof\; the\; variance\; of\\ & =a\frac{std(x_i)}{std(y_i)}(The\; linear\; coefficientamultiplied\; by{x}_{i}Japanese{\_}_{i}The\; ratio\; of\; the\; standard\; deviation\; of\end{array}$

For $R^2$ Derivation from top to bottom:
(4.43)~(4.45) Proof:$\begin{array}{r}{R}^2=1-\frac{RSS}{TSS}=1-\frac{MSE}{wa{r}_y}\end{array}$
(4.46)~(4.54)证明：$\begin{array}{r}{R}^2=1-\frac{RSS}{TSS}=a\frac{std(x_i)}{std(y_i)}\end{array}$
注：$y^{\prime }=a{x}_i+b$代入(4.48)，$b=\overline{y}-a\overline{x}$Substitute (4.49)

$\begin{array}{rlr}{R}^2& =1-\frac{RSS}{TSS}& (4.43)\\ & =1-\frac{\frac{1}{m}RSS}{\frac{1}{m}TSS}& (4.44)\\ & =1-\frac{MSE}{wa{r}_y}& (4.45)\\ & =\frac{TSS-RSS}{TSS}& (4.46)\\ & =\frac{ESS}{TSS}& (4.47)\\ & =\frac{\sum _{i=1}^m{\left(y^{\prime }-\overline{y}\right)}^2}{\sum _{i=1}^m{\left(y_i-\overline{y}\right)}^2}& (4.48)\\ & =\frac{\sum _{i=1}^m{\left(a{x}_i+b-\overline{y}\right)}^2}{\sum _{i=1}^m{\left(y_i-\overline{y}\right)}^2}& (4.49)\\ & =\frac{\sum _{i=1}^m{\left(a{x}_i+\left(\overline{y}-a\overline{x}\right)-\overline{y}\right)}^2}{\sum _{i=1}^m{\left(y_i-\overline{y}\right)}^2}& (4.50)\\ & =\frac{\sum _{i=1}^m{\left(a{x}_i-a\overline{x}\right)}^2}{\sum _{i=1}^m{\left(y_i-\overline{y}\right)}^2}& (4.51)\\ & =\frac{a^2\sum _{i=1}^m{\left(x_i-\overline{x}\right)}^2}{\sum _{i=1}^m{\left(y_i-\overline{y}\right)}^2}& (4.52)\\ & =a^2\frac{\frac{1}{m}\sum _{i=1}^m{\left(x_i-\overline{x}\right)}^2}{\frac{1}{m}\sum _{i=1}^m{\left(y_i-\overline{y}\right)}^2}& (4.53)\\ & =a^2\frac{var\left(x_i\right)}{var\left(y_i\right)}& (4.54)\end{array}$

5 Reference:

Machine Learning-Zhou Zhihua-P54
Language Sparrow Latex: https://www.yuque.com/yuque/gpvawt/brzicb
Symbol Katex: https://katex.org/docs/supported.html

Small bonus: For the derivation of the formula in part 4, you can click this link to download the relevant Markdown document.

---

2022-12-17 Add a problem transformation process to better understand the whole process of problem solving.
Question conversion clue:
Is (10,27) reasonable
-> find $f\left(x_i\right)=a{x}_i+The\; coefficient\; a\; and\; intercept\; b\; of\; b$ (a and b reach the optimal solution);
-> Find the mean square error$MSE=\frac{1}{m}{\sum }_{i=1}^m{\left(f\left(x_i\right)-y_i\right)}^{The\; value\; of\; a\; and\; b\; when\; 2}$ is the smallest;
-> Take the derivative of a and b respectively, take$\frac{\partial }{\partial a}{E}_{(a,b)}=0，\frac{\partial }{\partial b}{E}_{(a,b)}=0$ , solve the values ​​of a and b;
-> substitute$f\left(x_i\right)=a{x}_i+b$ , solution$x_i=The\; function\; value\; of\; 10$ is compared with 27.