困惑我好久的Multivariate Linear Regression 求导问题

Suppose we have ( $\vec{x}^{(i)}, y^{(i)}$) with sample size $N$, where $\vec{x}^{(i)} \in \mathbb{R}^D$.
$\hat{y} =\sum_{j=1}^D \beta_jx_j$
$\mathcal{L}(a, b)=\frac{1}{2}(a-b)^2$
$\begin{aligned} \varepsilon(\beta_0, \beta_1,..., \beta_{D}) &= \frac{1}{N}\sum_{i=1}^{N}\mathcal{L}(\hat{y}^{(i)}, y^{(i)}) \\ &= \frac{1}{2N} \sum_{i=1}^{N}(\hat{y}^{(i)}, y^{(i)})^2 \\ &=\frac{1}{2N} \sum_{i=1}^N (\sum_{j=1}^D \beta_j x_j^{(i)} -y^{(i)})^2 \end{aligned}$

Take Derivative with respect to $w_j$:
$\begin{aligned} \frac{\partial \varepsilon}{\partial \beta_j} &= \frac{1}{N} \sum_{i=1}^N x_j^{(i)}(\hat{y}^{(i)} -y^{(i)}) \\ &=\frac{1}{N} \sum_{i=1}^N x_j^{(i)}(\sum_{j'=1}^D \beta_{j'}x_{j'}^{(i)} -y^{(i)})(这部分注意：你就是这里不明白) \\ &= \frac{1}{N} \sum_{j'=1}^D (\sum_{i=1}^{N} x_j^{(i)} x_{j'}^{(i)})\beta_{j'} - \frac{1}{N}\sum_{i=1}^Nx_j^{(i)}y^{(i)} \end{aligned}$
Let $A_{jj'}=\frac{1}{N} \sum_{i=1}^N x_j^{(i)}x_{j'}^{(i)} \in \mathbb{R}^D$ and $c_j = \frac{1}{N}\sum_{i=1}^N x_j^{(i)}y^{(i)} \in \mathbb{R}^D$. Then:
$\begin{aligned} \frac{\partial \varepsilon}{\partial \beta_j} &= \frac{1}{N} \sum_{j'=1}^D (\sum_{i=1}^{N} x_j^{(i)} x_{j'}^{(i)})\beta_{j'} - \frac{1}{N}\sum_{i=1}^Nx_j^{(i)}y^{(i)} \\ &=\frac{1}{N}\sum_{j'=1}^D A_{jj'}\beta_{j'} -c_j \stackrel{set}{=}0 \end{aligned}$
Let $X \in \mathbb{R}^{N \times D}$ , $A=\frac{1}{N}X^TX$ and $c = \frac{1}{N} X^Ty$
$X= \left[ \begin{matrix} x^{(1)T} \\ x^{(2)^T}\\ .\\ .\\ x^{(n)^T} \end{matrix} \right] \tag{3}$
$\begin{aligned} \frac{\partial \varepsilon}{\partial \beta_j} &= \frac{1}{N} \sum_{j'=1}^D (\sum_{i=1}^{N} x_j^{(i)} x_{j'}^{(i)})\beta_{j'} - \frac{1}{N}\sum_{i=1}^Nx_j^{(i)}y^{(i)} \\ &=\frac{1}{N}\sum_{j'=1}^D A_{jj'}\beta_{j'} -c_j \\ &=A\beta-c \stackrel{set}{=} 0 \end{aligned}$
$\hat{\beta} = A^{-1}c = (X^TX)^{-1}X^Tt$
终于解决了！

一种更简单的方法是直接在risk做变换：
$\begin{aligned} \varepsilon(\beta_0, \beta_1,..., \beta_{D}) &=\frac{1}{2N} \sum_{i=1}^N (\sum_{j=1}^D \beta_j x_j^{(i)} -y^{(i)})^2 \\ &= \frac{1}{2N}[X\beta-y]^T [X\beta-y] \end{aligned}$
Finally, the MLE estimate is $\hat{\beta} = (X^TX)^{-1}X^Ty$

This is only a estimate from one single training data, but we really want to get the true error or prediction error, which can be defined as:
$\begin{aligned} \varepsilon_{true}(\beta_0, \beta_1,..., \beta_{D}) &=\frac{1}{2} E (\sum_{j=1}^D \beta_j \mathbf{x}_j - \mathbf{y})^2 \\ &= \frac{1}{2} \int_\mathbf{x}(\sum_{j=1}^D \beta_j \mathbf{x}_j - \mathbf{y})^2 p(\mathbf{x})d\mathbf{x} \end{aligned}$
If want to read more about bias-variance in linear regression model, read the following:
https://courses.cs.washington.edu/courses/cse546/12wi/slides/cse546wi12LinearRegression.pdf