[Linear Algebra/Machine Learning] Singular values of matrices and singular value decomposition (SVD)

I. Introduction

We know that for a $n\times$matrixAA$of\; n$$A$ , Rugo$A$ have$n$ linearly independent feature vectors, then$A$ can be similarly diagonalized, that is, there is an invertible matrixPPLet$A$$A=P\Lambda P^{-1}$ , where$\Lambda$ isAAA diagonal matrix composed of the eigenvalues ​​of$A.$$The\; column\; of\; P$ is actuallyAAEigenvector of$A.$Put$A$ decomposes into$P\Lambda P^{The\; process\; of\; -1}$ is called eigendecomposition of the matrix. However, for$m\times Matrix\; of\; n$ , where$m\ne n$ , there is nothing we can do. How should we decompose this matrix at this time? Here we introduce the concept of singular value decomposition (SVD).

2. Singular value

设$A$ is a$m\times n$ matrix. We are already familiar with eigenvalues, so our definition of singular values ​​is also derived from eigenvalues. What kind of matrix has eigenvalues? The answer is square array. But$A$ is not necessarily a square matrix, but we have a way to turn it into a square matrix -$A^{T}A$is$n\times n$ square matrix. Next we examine$A^{}$Eigenvalues ​​of${}^T$$A.$

Logic 1 $A^{}$Each eigenvalueλ \lambda${}^{of\; T}$$A$$\lambda$ are all greater than or equal to$0$。

Proof : Suppose $A^{T}A\mathbf{x}=\lambda x$ , where$\mathbf{x}$是$A^{}$An eigenvector of${}^T$$A.$Then$$\begin{gathered} {\mathbf{x}}^T{A}^{T}Ax=\lambda {\mathbf{x}}^T\mathbf{x} \\ \Vert A\mathbf{x}{\Vert }^2=\lambda \Vert \mathbf{x}{\Vert }^{2Note} \end{gathered}$$ that$\Vert Ax{\Vert }^2$ and$\Vert \mathbf{x}{\Vert }^2$ are both non-negative numbers, so$l\ge 0$。∎

Now let's define singular values.

Enclosure 2 $l_1,l_2,\cdots ,l_n$是$A^{T}A$的奇异值，使用$l_1\ge l_2\ge \cdots \ge l_n\ge 0$ if$p_i=\sqrt{l_i}$，则$p_1\ge p_2\ge \cdots \ge p_n\ge 0$ _$p_i$called AAThe singular value of$A.$

Mission 3 AAThe number of non-zero singular values ​​of$A$$A$ 's rank.

Proof : Immediate confirmation $r(A^{T}A)=r\left(A\right)$ . Consider the system of homogeneous linear equations$A^{T}A\mathbf{x}=0$ , let$\xi$ is one of its solutions, that is,$A^{T}A\mathbf{\xi }=0$，则$X^T{A}^{T}A\xi =0$，ie∥$\Vert A\mathbf{\xi }{\Vert }^2=0$，故$A\mathbf{\xi }=0$ . This shows that$A^{T}Ax=$The solution to$0$$A\mathbf{x}=0$ solution. At the same time$Ax=The\; solution\; to\; 0$ is obviously$A^{T}A\mathbf{x}=0$ solution, so the two linear equations have the same solution, which means that$r(A^{T}A)=r(A)$。∎

Many times we will encounter such a problem: $\Vert x\Vert$ and∥ A x ∥ \|A\boldsymbol{x}\|What is the relationship between the size of$\Vert$$A$$x$$\Vert \; ?$Put matrix$A$ is regarded as a linear transformation, which acts on$The\; length\; of\; x$ can be changed, so how many times can the length change at most? With singular values, we can easily solve this problem.

Proposition 4 Let $A$ is a$m\times n$ matrix,$x$ is an$n\times 1$ vector. Then$\Vert A\mathbf{x}\Vert \le p_1\Vert x\Vert$ , where$p_1$Yes $The\; largest\; singular\; value\; of\; A$ , and the equality condition is$\mathbf{x}$是$A^{T}A$corresponds to the eigenvalue$p_1^2$eigenvector.

Proof : Note $A^{T}A$is a real symmetric matrix, so it has unit orthogonal eigenvector groups${\mathbf{v}}_1,v_2,\cdots ,v_n$。若$\mathbf{x}\in {\mathbb{R}}^n$ , you can put$x$ is expressedas$$x=c_1{v}_1+c_2{v}_2+\cdots +c_n{v}_n$$Among them $c_1,c_2,\cdots ,c_n$is a scalar, satisfying $c_1^2+c_2^2+\cdots +c_n^2=\Vert \mathbf{x}{\Vert }^2$ . Examine again$\Vert A\mathbf{x}{\Vert }^2$：$$\Vert A\mathbf{x}{\Vert }^2={\mathbf{x}}^T{A}^{T}A\mathbf{x}=⟨\mathbf{x},A^{T}A\mathbf{x}⟩=⟨\sum _{i=1}^n{c}_i{\mathbf{v}}_i,\sum _{i=1}^n{c}_i{A}^{T}A{\mathbf{v}}_i⟩$$ note$v_i$是$A^{T}A$corresponds to the eigenvalue$p_i^2$special direction, so $A^{T}A{\mathbf{v}}_i=p_i^2{\mathbf{v}}_i$。因此$$\Vert A\mathbf{x}{\Vert }^2=⟨\sum _{i=1}^n{c}_i{v}_i,\sum _{i=1}^n{c}_i{p}_i^2{v}_i⟩=\sum _{i=1}^n{c}_i^2{p}_i^2\le \sum _{i=1}^n{c}_i^2{p}_1^2=p_1^2\Vert \mathbf{x}{\Vert }^{}$$The equal condition for$${}^2$$$c_1^2=\Vert x{\Vert }^2$且$c_2=c_3=\cdots =c_n=0$ , at this time$\mathbf{x}=c_{1}v$ , so$\mathbf{x}$是$A^{T}A$corresponds to the eigenvalue$p_1^2$eigenvector. Certification completed. ∎

If $x\perp {\mathbf{v}}_1$, that is, $c_1=0$ , then it can be proved similarly$\Vert A\mathbf{x}\Vert \le p_2\Vert \mathbf{x}{\Vert }^2$ ; if$x\perp v_1$And $\mathbf{x}\perp {\mathbf{v}}_2$, that is, $c_1=c_2=0$，则$\Vert A\mathbf{x}\Vert \le p_3\Vert \mathbf{x}{\Vert }^2$ ; and so on.

3. Definition of singular value decomposition

Singular values ​​have been introduced above. Next, we will introduce how to use singular values ​​to decompose matrices.

设$A$ is a$m\times n$矩阵，$p_1\ge p_2\ge \cdots \ge p_n\ge 0$ is its singular value. Let$r$为$The\; rank\; of\; A$ , which isAAThe number of non-zero singular values ​​of$A.$

Definition 5 $A\; singular\; value\; decomposition\; of\; A$ is a decomposition of the form:$$A=U\Sigma V^T$$ among them

• $U$ is a$m\times m$ orthogonal matrix;
• $V$ is an$n\times northogonal$ matrix;
• $\Sigma$ is a$m\times n$ matrix, which is very similar to a diagonal matrix: its$The\; i$ diagonal elements are$p_i$, for $i=1,2,\cdots ,r$。 Σ \Sigma The other elements of$\Sigma$$0$。

For example, when $When\; A$ is a symmetric square matrix, its singular values ​​are actually the absolute values ​​of the eigenvalues.

4. How to perform singular value decomposition

引理6
(1) $\Vert A{\mathbf{v}}_i\Vert =p_i$；
(2) 若$i\ne j$，则$A{v}_i$with $A{v}_j$orthogonal.

proof：$⟨A{v}_i,A{v}_j⟩={\mathbf{v}}_i^T{A}^{T}A{\mathbf{v}}_j=v_i^T{p}_j^2{v}_j=p_j^2⟨{\mathbf{v}}_i,v_j⟩$。

• Waka $i=j$，由$\Vert {\mathbf{v}}_i\Vert =1$知$\Vert A{\mathbf{v}}_i{\Vert }^2=p_i^2$；
• 若$i\ne j$，由${\mathbf{v}}_i\perp v_j$知$A{v}_i\perp A{v}_j$。

Theorem 7 Let $A$ is a$m\times n$ matrix. Then we can construct anAASingular value decomposition of$A$$A=U\Sigma V^T$ , where:

• $V$是$A^{The}unit$orthogonal eigenvector group${\mathbf{v}}_1,v_2,\cdots ,v_n$，满足$A^{T}A{\mathbf{v}}_i=p_i^2{v}_i$；
• 若$i\le r$ (at this time$p_i\ne 0$ )，则$U$ part$i$列是$\frac{1}{p_1}A{v}_i$. According to Lemma 6, these columns are unit orthogonal, and other columns can be arbitrarily expanded by ${\mathbb{R}}^{}$The unit orthonormal basis of${}^{m\; is\; obtained.}$

Proof : We only need to prove that if $UwaVV$ _$V$ is defined as above, then$A=U\Sigma V^{T.}$ _ We cannot directly prove that$A=U\Sigma V^T$ , but we can prove that$\forall \mathbf{x}\in {\mathbb{R}}^n$，$U\Sigma V^T\mathbf{x}=Ax$ .$\_$This can show that$A=U\Sigma V^T$ (because if$\forall x\in R^n$ has$A\mathbf{x}=B\mathbf{x}$，则$\forall \mathbf{x}\in {\mathbb{R}}^n$ has$(A-B)\mathbf{x}=0$ , that is, the rank of the basic solution system of this linear equation system is$n$，$r(A-B)=0$，$A-B=O$，$A=B$）。考虑$$V^T\mathbf{x}=\left[\begin{array}{c}{\mathbf{v}}_1^T\\ v_2^T\\ ⋮\\ v_n^T\end{array}\right]\mathbf{x}=\left[\begin{array}{c}{v}_1^{T}x\\ v_2^{T}x\\ ⋮\\ v_n^T\mathbf{x}\end{array}\right]$$则$$\Sigma V^{T}x=\left[\begin{array}{c}{p}_1{v}_1^{T}x\\ p_2{v}_2^{T}x\\ ⋮\\ p_r{v}_r^{T}x\\ 0\\ ⋮\\ 0\end{array}\right]$$Left side $U$ profit$$\begin{array}{rl}U\Sigma V^T\mathbf{x}& =(p_1{\mathbf{v}}_1^T\mathbf{x})\frac{1}{p_1}A{v}_1+\left(p_2{v}_2^{T}x\right)\frac{1}{p_2}A{v}_2+\cdots +\left(p_r{v}_r^{T}x\right)\frac{1}{p_r}A{v}_r\\ & =A{v}_1{v}_1^T\mathbf{x}+A{v}_2{v}_2^{T}x+\cdots +A{v}_r{v}_r^{T}x\\ & =A{v}_1{v}_1^{T}x+A{v}_2{v}_2^{T}x+\cdots +A{v}_r{v}_r^{T}x+\cdots +A{v}_n{v}_n^{T}x\\ & =A({\mathbf{v}}_1{v}_1^T+v_2{v}_2^T+\cdots +v_n{v}_n^T)\mathbf{x}\\ & =A{V}^{T}Vx\\ & =A\mathbf{x}\end{array}$$Note that when i > r i>r is used here$i>r$时$A{v}_i=p_i=0$ . This proves that$A=U\Sigma V^T$。∎

The singular value decomposition of matrices is widely used in machine learning, such as playing an important role in Principal Component Analysis (PCA).

References

1. https://en.wikipedia.org/wiki/Eigendecomposition_of_a_matrix
2. https://math.berkeley.edu/~hutching/teach/54-2017/svd-notes.pdf