投影矩阵和最小二乘

一个最小二乘法的例子：

三个点分别是 $(1,1), (2,2),(3,2)$. 对这三个点进行回归分析，假设以下的方程：

$$ y = C + D t$$

那么有矩阵运算：

$$\begin{bmatrix} 1& 1 \\ 1 & 2 \\1 & 3 \end{bmatrix}\begin{bmatrix} C\\D \end{bmatrix} = \begin{bmatrix} 1 \\ 2 \\2 \end{bmatrix}$$

显然这个方程是不可解的。最小二乘法求解最优值的方案在于

$$\Vert Ax-b \Vert ^2 = \Vert e \Vert ^2 = e^2_1+e^2_2+e^2_3$$

---

At last, if we have the matrix $A$ which has independent columns, then $A^TA$ is invertible. Or we can say, suppose $A^TAx = 0$, $x$ must be $0$.

Proof:  $$x^TA^TAx = 0  \to (Ax)^TAx = 0$$

then we have $Ax=0$. Considering that $A$ has independent columns, $x$ must be $0$.