[Linear Algebra 04] Projection matrix P and standard orthogonal matrix Q

  Continuing with the content of the MIT notes, the solution of AX=b was discussed earlier, and I focused on drawing some graphics of the solution space. Later we divided these subspaces into four parts, namely AAThe column space, row space and null space of$A$$A^{}$For the null space of${}^T$ , the relationship between these four spaces is shown in figure 4.2 of the textbook. In fact, we can find such a vertical relationship from the last example last time. It should be pointed out that the vertical relationship is a very good relationship. This time we mainly introducetheprojection matrix Pandthe standard orthogonal matrix Q.

“Follow the rules”

  When the old man talked about orthogonal vectors and orthogonal subspaces, he asked whether the zero vector is orthogonal to any vector. He gave such a suggestion " The one thing about math is you're supposed to follow the rules. " This The flavor of the suggestion is that when everything is difficult to decide or even unbelievable, you should return to the definition. If the definition allows it to happen and it happens, then it is reasonable and you should face it squarely. So, whether the zero vector is orthogonal to any vector, the answer is "sure". So let's look at the definition of orthogonal (or vertical in spatial relation):

If the inner product of two vectors in the inner product space is 0 , then they are orthogonal .

  Taking figure4.2 as an example, the known null space is defined as

The null space appears in the context of a linear map (that is, a matrix), and refers to the original image space of zero, that is, { x ∣ A x = 0 x| Ax=0 $x|Ax=0$} 。

  It can be seen that the definition of the null space happens to be embedded with the definition of orthogonality . So see $AX=0$，$A$ throughXXNull space represented by$X$ and AAThe inner product of the row space represented by$A$$A$ 's Null Space and$The\; line\; space\; of\; A$ is vertical, that is, the left part of figure4.2; look at$A^{T}Y=0$，$A^T$ through$Null\; space\; represented\; by\; Y$ and$A^{The\; row\; space\; of\; T}$ (due to the transposition, i.e.$The\; column\; space\; of\; A$ ) vertically, shown as the right part of figure 4.2.

Projection matrix P

   What is a projection matrix? We illustrate this concept in conjunction with figure 4.6.

Start with vectors

   Look at the left part first, this is a vector $\vec{b}$projected onto $\vec{a}$Combining with the definition of orthogonality, we have
$$\vec{e}\cdot \vec{a}=0\Rightarrow (\vec{b}-\vec{p})\cdot \vec{a}=0\Rightarrow \vec{b}\cdot \vec{a}-\vec{p}\cdot \vec{a}=0\Rightarrow \vec{p}\cdot \vec{a}=\vec{b}\cdot \vec{a}$$
   Multiply both sides by $\vec{a}$, Note that vector multiplication does not satisfy the associative law, but matrix multiplication does, but matrix multiplication does not satisfy the commutative law, but vector multiplication does. Then there is
$$\vec{p}|a{|}^2=\vec{b}\cdot \vec{a}\cdot \vec{a}\Rightarrow \vec{p}=\frac{\vec{b}\cdot \vec{a}}{|a{|}^2}\vec{a}=\frac{\vec{a}\cdot \vec{b}}{|a{|}^2}\vec{a}$$
   We express the above vector formula in matrix form, $a^{T}a$means for$\vec{a}$The square of the modulus of the column vector represented, and note that $a^{T}b$也为一数，就有：
$$p=\frac{a^{T}b}{a^{T}a}a=a\frac{a^{T}b}{a^{T}a}=\frac{a{a}^T}{a^{T}a}The\; b$$
   projection matrix then plays such a role and finds out the vector$\vec{b}$In the vector $\vec{a}$On the component, that is, $p=Pb=a\hat{x}$ . For the left part, the projection matrix is
$$P=\frac{a{a}^T}{a^{T}a}$$

Look at the matrix again

   But looking at this process, we can easily find that $a^{T}a$ 和 $a^{The\; condition\; that\; T}b$is a number is harsh, and it is aimed at a one-dimensional column matrix; for general matrices, the results of these two are matrices rather than a number. So we continue to look at the right part and project b onto a plane represented by a two-dimensional column vector. Although the equation has become more complicated, the obtained equation is still the same, that is, using a set of bases in the plane and$\vec{e}$The result of the inner product is 0. Then for $As\; far\; as\; A$ is concerned, the plane where its two bases are located is its column space. If e is regarded as a column vector, the equation can be solved:
$$A^{That\text{'}s}it=0$$
   We generate e from the perspective of the projection matrix P:
$$e=b-p=b-Pb=b-A\hat{x}$$
   反代后，就有:
$$A^T(b-A\hat{x})=A^{T}b-A^{T}A\hat{x}=0\Rightarrow A^{T}A\hat{x}=A^{T}b\Rightarrow \hat{x}=(A^{T}A{)}^{-1}{A}^{T}b\Rightarrow p=A\hat{x}=A(A^{T}A{)}^{-1}{A}^{T}b$$
   finally deduces the projection matrix P
$$P=A(A^{T}A{)}^{-1}{A}^{But}$$
   this derivation is obviously based on an assumption that"If A has independent columns, then$A^{T}A$is invertible."In other words, for A, there is a left invert matrix.

least square method

   An interesting usage of the projection matrix is ​​to derive the least squares method, which is often used in linear regression. We can either use the general gradient descent method to give the computer iterative derivatives to solve, or directly start from the matrix. Let's look at a general example:
$$Ax=\left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\\ a_{31}& a_{32}\\ .& .\\ .& .\\ .& .\end{array}\right]\left[\begin{array}{c}C\\ D\end{array}\right]=\left[\begin{array}{c}{b}_1\\ b_2\\ b_3\\ .\\ .\\ .\end{array}\right]=b$$
   $Ax=b$ When the number of data points (number of equations) is much greater than the number of fitting parameters (2 unknowns), it is obvious that there is either a solution or no solution. According to the idea of ​​least squares, when faced with no solution, we should seek the minimum value of the following function, namely
$$min(a_{11}C+a_{12}D-b_1{)}^2+(a_{21}C+a_{22}D-b_2{)}^2+(a_{31}C+a_{32}D-b_3{)}^2+...We$$
  compare C sum D partial deviation (abbreviated coefficient 2), with
$$\begin{gathered} a_{11}(a_{11}C+a_{12}D-b_1)+a_{21}(a_{21}C+a_{22}D-b_2)+a_{31}(a_{31}C+a_{32}D-b_3)+...=0 \\ {a}_{12}(a_{11}C+a_{12}D-b_1)+a_{22}(a_{21}C+a_{22}D-b_2)+a_{32}(a_{31}C+a_{32}D-b_3)+...=0 \end{gathered}$$
  We add the two formulas together, and it is not difficult to find that the coefficient of C is $\sum a_{i1}^2+a_i{a}_{i2}$, the D-like coefficient is $\sum a_{i2}^2+a_i{a}_{i2}$, and $b_i$The former coefficient is equivalent to $a_i+a_{i2}$. So we can write this equation
$$A^{T}A\left[\begin{array}{c}C\\ D\end{array}\right]=A^{}$$
  When$${}^T$$$$b$$
$$\left[\begin{array}{c}C\\ D\end{array}\right]=(A^{T}A{)}^{-1}{A}^{T}b$$
  Then the question arises, how to derive this formula from P? We know thatwhen P exists, we can project b, which is not in the column space of A, into the column space of A, which can form a solution guarantee. That is to say:
$$Pb=A\hat{x}\Rightarrow A(A^{T}A{)}^{-1}{A}^{T}b=A\hat{x}$$
  So we have
$$\hat{x}=(A^{T}A{)}^{-1}{A}^{T}b$$
  From this point of view,∑ ei \sum e_igenerated by projecting the vector onto the plane$\sum e_i$(vector sum) min . We can use matlab to verify its correctness, that is, first use the polyfit function to solve it, and then use the matrix to verify it .

% 用matlab自带的ployfit函数拟合曲线

x=[9,13,15,17,18.6,20,23,29,31.7,35];
y=[-8,-6.45,-5.1,-4,-3,-1.95,-1.5,-0.4,0.2,-0.75];
coefficient=polyfit(x,y,1);  %用一次函数拟合曲线，想用几次函数拟合，就把n设成那个数
y1=polyval(coefficient,x);
plot(x,y,'o',x,y1,'-');
legend('散点',['拟合曲线y=',num2str(coefficient(1)),'*x',num2str(coefficient(2))]);

% 验证
c = ones(1,length(x)); % 常数列
A = [c;x]'; % 矩阵A
est = inv(A'*A)*A'*y'; % 估计值
text(15,-6,['常数项系数=',num2str(est(1)),'  一次项系数=',num2str(est(2))]);

orthonormal matrix

nature

   The so-called standard means that the modulus value of each vector in the vector group that makes up the matrix is ​​1, and the so-called orthogonal means that the inner product of any two vectors in the vector group is 0, which is the "Follow the rules" reminded at the beginning . It is stipulated that such a special matrix must have some excellent properties. Since the inner product of any two vectors is 0, we can easily understand that they are all linearly independent. According to the definition of standard orthogonal matrix, we can easily deduce the following formula:
$$Q{Q}^T=IBecause$$ each column of Q is only related to itself, and the modulus is 1, and it is irrelevant to the other columns, and the inner product is 0, so the result will be an identity
   matrix. Further, for the square matrix, we know that since the columns are linearly independent, the inverse must exist, and the definition of the inverse seems to be hereQTQ^T$Q^{The\; position\; of\; T}$ , so, there is
$$Q^{-1}=Q^{The\; inverse\; of\; the\; T}$$
   orthonormal matrix is ​​its transpose, a gift that makes people happy enough.

process

  Let's show the method of standard orthogonalization of the matrix, that is, Gram-Schmidt orthogonalization. This step is divided into two steps, the first step is orthogonalization, and the second step is normalization. To understand the principle of orthogonalization, we can also refer to the graph of our projection matrix P, that is, instead of seeking p, we are seeking the orthogonal e .

  With the help of figure4.6, we know that e should be obtained like this:
$$B=b-Pb=b-a(a^{T}a{)}^{-1}{a}^{T}b=b-\frac{a^{T}b}{a^{T}a}aSimilarly$$
  ,if it is for a plane, the two bases that have been orthogonalized before should be subtracted, namely:
$$C=c-P_{1}c-P_{2}c=c-\frac{a^{T}c}{a^{T}a}a-\frac{B^{T}c}{B^{T}B}B$$

  可以验证$a^{T}B=0，a^{T}C=0，B^{T}a=0，B^{T}C=0$ , which shows the correctness of our orthogonalization method.

  According to the diagram, let’s focus on the process of orthogonalization of column 3. As we know from the above, the two items subtracted from c are actually the projections of c on a and B. If we add this projection first, the result is blue The red diagonal line ON in the color plane is equivalent to explaining:
$$column3-redcrossingline\_=Orthogonalizationofcolumn3?\_\Rightarrow MN\perp PlanarOPNQ?$$

  Actually, this conclusion is obvious:
$$\begin{gathered} OP\perp PM，OP\perp PN(OQ)\Rightarrow OP\perp MN \\ OQ\perp QM，OQ\perp QN\left(OP\right)\Rightarrow OQ\perp MN \\ MN\perp OP,MN\perp OP\Rightarrow MN\perp PlaneOPNQ \end{gathered}$$
  From this, it can be reasonably extrapolated that for the hyperplane, the nth vector should subtract the n-1 bases that have been orthogonalized . Of course, this kind of perspective based on the vertical starting point of space is cumbersome for the matrix, and $A=Like\; the\; LU$ transformation, the standard orthogonalization of the matrix also corresponds to a matrix R, that is,$A=QR$ . We can easily see that this R is an upper triangular matrix (to verify the orthonormal property of the matrix Q), take the three-dimensional space as an example
$$\left[\begin{array}{ccc}a& b& c\end{array}\right]=\left[\begin{array}{ccc}{q}_1& q_2& q_3\end{array}\right]\left[\begin{array}{ccc}{q}_1^{T}a& q_1^{T}b& q_1^{T}c\\ 0& q_2^{T}b& q_2^{T}c\\ 0& 0& q_3^{T}c\end{array}\right]$$
  Deduce $q_1$、$q_2$、$q_3$The process of is still one step away, that is, standardization, just divide the orthogonalized vector by its modulus :
$$q_1=\frac{A}{|A|}{q}_2=\frac{B}{|B|}{q}_3=\frac{C}{|C|}$$

example

  To end with an example:
$$A=\left[\begin{array}{ccc}1& 3& 1\\ 2& 2& 2\\ 3& 1& 1\end{array}\right]=\left[\begin{array}{ccc}{a}_1& a_2& a_3\end{array}\right]$$

  Let’s take a look at the space first:

% 画向量图
quiver3(0,0,0,1,2,3,'m'); hold on; quiver3(0,0,0,3,2,1,'black'); % 基底
hold on;  quiver3(0,0,0,1,2,1,'r');  % 列3
V1 = [1;2;3]; V2 = [3;2;1];

% 求法向量
Vn = cross(V1,V2);
Vn = Vn/norm(Vn);

% 画单位法向量
hold on;quiver3(0,0,0,Vn(1),Vn(2),Vn(3),'g');
% 画平面
syms x1;syms x2;syms x3;
plane = -(x1*Vn(1)+x2*Vn(2))/Vn(3);
hold on;
fmesh(plane);
% 标注
legend('列1','列2','列3','单位法向量','列1和列2所在平面');

  首先进行正交化，即
$$\begin{gathered} A=a_1=\left[\begin{array}{c}1\\ 2\\ 3\end{array}\right] \\ B=a_2-\frac{A^T{a}_2}{A^{T}A}A=\left[\begin{array}{c}3\\ 2\\ 1\end{array}\right]-\frac{10}{14}\left[\begin{array}{c}1\\ 2\\ 3\end{array}\right]=\left[\begin{array}{c}16/7\\ 4/7\\ -8/7\end{array}\right] \\ C=a_3-\frac{A^T{a}_3}{A^{T}A}A-\frac{B^T{a}_3}{B^{T}B}B=\left[\begin{array}{c}1\\ 2\\ 1\end{array}\right]-\frac{8}{14}\left[\begin{array}{c}1\\ 2\\ 3\end{array}\right]-\frac{112}{336}\left[\begin{array}{c}16/7\\ 4/7\\ -8/7\end{array}\right]=\left[\begin{array}{c}-1/3\\ 2/3\\ -1/3\end{array}\right] \end{gathered}$$
  Then normalize,
$$\begin{gathered} q_1=\frac{A}{|A|}=\left[\begin{array}{c}1\\ 2\\ 3\end{array}\right]/\sqrt{14}=\left[\begin{array}{c}0.2673\\ 0.5345\\ 0.8018\end{array}\right] \\ {q}_2=\frac{B}{|B|}=\left[\begin{array}{c}16/7\\ 4/7\\ -8/7\end{array}\right]/\sqrt{336/49}=\left[\begin{array}{c}0.8729\\ 0.2182\\ -0.4362\end{array}\right] \\ {q}_3=\frac{C}{|C|}=\left[\begin{array}{c}-1/3\\ 2/3\\ -1/3\end{array}\right]/\sqrt{6/9}=\left[\begin{array}{c}-0.4082\\ 0.8165\\ -0.4082\end{array}\right] \end{gathered}$$

  See below for the programming implementation of the above-mentioned Schmidt orthogonalization method. The orth function that can also be included in matlab can directly find a set of standard orthogonal bases , and can verify the correctness of the results.

a = [1,3,1;2,2,2;3,1,1];

%% 施密特正交化方法
[m,n] = size(a);
if(m<n)
    error('行小于列，无法计算，请转置后重新输入');
end
b=zeros(m,n);
%正交化
b(:,1)=a(:,1);
for i=2:n
    for j=1:i-1
        b(:,i)=b(:,i)-dot(a(:,i),b(:,j))/dot(b(:,j),b(:,j))*b(:,j);
    end
    b(:,i)=b(:,i)+a(:,i);
end

%单位化
for k=1:n
    b(:,k)=b(:,k)/norm(b(:,k));
end

%% 直接用orth方法
result = orth(a);
b
result

b =

    0.2673    0.8729   -0.4082
    0.5345    0.2182    0.8165
    0.8018   -0.4364   -0.4082


result =

   -0.5494   -0.7071   -0.4451
   -0.6295   -0.0000    0.7770
   -0.5494    0.7071   -0.4451