[Mathematical knowledge] Least squares method, general linear situation, matrix representation process, optimal parameter solution formula process

serial number	content
1	[Mathematical knowledge] Degree of freedom and calculation method of degrees of freedom
2	[Mathematical knowledge] Rigid body rigid body and the motion of rigid body
3	[Mathematical knowledge] Basic motion of rigid bodies, translation and rotation
4	[Mathematical knowledge] Vector multiplication, inner product, outer product, matlab code implementation
5	[Mathematical knowledge] Covariance, the covariance of random variables, the covariance when the random variables are single numbers and vectors respectively
6	[Mathematical knowledge] The derivation process of the rotation matrix is ​​based on the rotation of the vector, while solving the nonlinear limitations of the Euclidean transformation.

Friendly reminder: Please read the article [Mathematical Knowledge] Least Squares Method first, starting from linear regression, giving numerical examples and using the least squares method to solve the regression model , and then read this article.

In the previous article [Mathematical Knowledge] Least Squares Method, starting from linear regression, giving numerical examples and using the least squares method to solve the regression model, starting from the perspective of regression analysis, the linear regression model was analyzed, specific numerical examples were given, and using The least squares method solves the optimal parameters of the model.

In the previous regression analysis, the example used was:

• Simple linear regression model (single explanatory variable, single response variable)
• Multiple linear regression model (multiple explanatory variables, single response variable)

However, they are all single response variables, and it is not convenient to carry out matrix description.

In practical applications, regression models with multiple explanatory variables and multiple response variables are more commonly used, and matrix operations are also used, which can not only save computer resources but also speed up calculation efficiency.

Therefore, this article will start from multiple explanatory variables and multiple response variables, focusing on how to matrix it, as well as the derivation operations and operations after matrixing.

---

1. Multiple explanatory variables and multiple response variables

On the basis of the above (single explanatory variable, single response variable) sum, it is extended to a more general linear case.

This kind of regression model with multiple explanatory variables and multiple response variables can be expressed by the formula as

$$\begin{array}{rl}{y}_1& =b_0+b_1{x}_{11}+b_2{x}_{12}+\cdots +b_p{x}_{1p}\\ y_2& =b_0+b_1{x}_{21}+b_2{x}_{22}+\cdots +b_p{x}_{3p}\\ y_3& =b_0+b_1{x}_{31}+b_2{x}_{32}+\cdots +b_p{x}_{3p}\\ ⋮& \\ y_m& =b_0+b_1{x}_m+b_2{x}_m+\cdots +b_p{x}_{mp}\end{array}$$

You can see that this regression model has a total of
$m$ response variables$y_1,\cdots ,y_m$，
$m\ast p$ explanatory variables$x_{11},\cdots ,x_{mp}$，
$1$ parameter$b_0$，
$p$ parameters$b_1,\cdots ,b_p$。

---

Next, matrixization is performed, so that

$$Y={\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_m\end{array}\right]}_{m\times 1},X={\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_m& x_m& \cdots & x_{mp}\end{array}\right]}_{m\times p},b={\left[\begin{array}{c}{b}_1\\ b_2\\ ⋮\\ b_p\end{array}\right]}_{p\times 1}$$

$$\begin{array}{r}Y=X\beta +b_0\end{array}$$

Although this form looks simple and clear, it has one more term $b_0$, we change $b_0$Integrate into $Go\; to\; \beta$ . make

$$Y={\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_m\end{array}\right]}_{m\times 1},X={\left[\begin{array}{ccccc}1& x_{11}& x_{12}& \cdots & x_{1p}\\ 1& x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 1& x_m& x_m& \cdots & x_{mp}\end{array}\right]}_{m\times (p+1)},b={\left[\begin{array}{c}{b}_0\\ b_1\\ b_2\\ ⋮\\ b_p\end{array}\right]}_{(p+1)\times 1}$$

Then it became

$$\begin{array}{r}Y=X\beta \end{array}$$

---

2. Matrix operations to find partial derivatives

The next focus is to find the best $\theta$ derivativeX$X\beta -Y$ is the smallest, that is

$$\underset{b}{\min }\Vert X\beta -Y{\Vert }_2^2$$

Among them $X,Y$ is known, the unknown is$b$ .

Use the least squares method to find the optimal parameter $When\; \beta$ , we often see a conclusion formula, that is, the optimal$\beta$ is

$$b=(X^{\text{T}}X{)}^{-1}{X}^{\text{T}}Y$$

But I still don’t know the solution process. Therefore, how to arrive at this conclusion will be discussed next.

---

令 $J\left(b\right)=\Vert X\beta -Y{\Vert }_2^2$, which is another way to express the sum of squares of the residuals. We can find its corresponding matrix form by expanding this equation as

$$J\left(b\right)=(X\beta -Y{)}^{\text{T}}(X\beta -Y)$$

Next calculate $J\left(\beta \right)$对β \betaThe partial derivative of$\beta$$The\; value\; of\; \beta$ . Expanding the above equation we have

$$\begin{array}{rl}J\left(b\right)& =(X\beta -Y{)}^{\text{T}}(X\beta -Y)\\ & =(b^{\text{T}}{X}^{\text{T}}-Y^{\text{T}})(X\beta -Y)\\ & =b^{\text{T}}{X}^{\text{T}}X\beta -b^{\text{T}}{X}^{\text{T}}Y-Y^{\text{T}}X\beta +Y^{\text{T}}Y\end{array}$$

To find the partial derivative, we have

$$\begin{array}{rl}\frac{\partial J(b}{\partial \beta }& =\frac{\partial }{\partial \beta }(b^{\text{T}}{X}^{\text{T}}X\beta -b^{\text{T}}{X}^{\text{T}}Y-Y^{\text{T}}X\beta +Y^{\text{T}}Y)\end{array}$$

To find out this formula, you must use the rules of matrix calculus: For the unknown quantity $X$ and constant matrix$A$ , yes

• $\frac{\text{d}}{\text{d}X}(AX)=A^{\text{T}}$
• $\frac{\text{d}}{\text{d}X}(X^{\text{T}}A)=A$
• $\frac{\text{d}}{\text{d}X}(X^{\text{T}}AX)=(A+A^{\text{T}})X$

According to the above rules, there are

• $\frac{\partial }{\partial \beta }(b^{\text{T}}{X}^{\text{T}}X\; \beta )=((X^{\text{T}}X)+(X^{\text{T}}X{)}^{\text{T}})b=2X{\_}^{\text{T}}X\beta$
• $\frac{\partial }{\partial \beta }(-b^{\text{T}}{X}^{\text{T}}Y)=-X^{\text{T}}Y$
• $\frac{\partial }{\partial \beta }(-Y^{\text{T}}X\; \beta )=(-Y^{\text{T}}X{)}^{\text{T}}=-X^{\text{T}}Y$

Therefore, the partial derivative is

$$\frac{\partial J(b}{\partial \beta }=2X{\_}^{\text{T}}X\beta -2X{\_}^{\text{T}}Y$$

Let it be zero

$$\begin{array}{rl}2X{\_}^{\text{T}}X\beta -2X{\_}^{\text{T}}Y& =0\\ X^{\text{T}}X\beta -X^{\text{T}}Y& =0\\ X^{\text{T}}X\beta & =X^{\text{T}}Y\\ & =(X^{\text{T}}X{)}^{-1}{X}^{\text{T}}Y\end{array}$$

At this point, the solution formula for the optimal parameters of the least squares method has been obtained.

---

Ref

1. Least squares method - WikiPedia
2. The least squares method, how to use it and how to derive it - bilibili
3. How to understand the least squares method? - Classmate Ma

---