7.4 The singular value decomposition (奇异值分解)

本文为《Linear algebra and its applications》的读书笔记

As we know, not all matrices can be factored as $A=PD{P}^{-1}$ with $D$ diagonal. However, a factorization $A=QD{P}^{-1}$ is possible for any $m\times n$ matrix $A$ ! A special factorization of this type, called the singular value decomposition, is one of the most useful matrix factorizations in applied linear algebra.

The singular value decomposition is based on the following property of the ordinary diagonalization that can be imitated for rectangular matrices: The absolute values of the eigenvalues of a symmetric matrix $A$ measure the amounts that $A$ stretches or shrinks certain vectors (the eigenvectors). If $A\mathbf{x}=\lambda \mathbf{x}$ and $\Vert \mathbf{x}\Vert =1$, then

If ${\lambda }_1$ is the eigenvalue with the greatest magnitude, then a corresponding unit eigenvector ${\mathbf{v}}_1$ identifies a direction in which the stretching effect of $A$ is greatest. This description of ${\mathbf{v}}_1$ and $\Vert {\lambda }_1\Vert$ has an analogue for rectangular matrices that will lead to the singular value decomposition.

EXAMPLE 1
If $A=\left[\begin{array}{ccc}4& 11& 14\\ 8& 7& -2\end{array}\right]$, then the linear transformation $\mathbf{x}\mapsto A\mathbf{x}$ maps the unit sphere(球) { x : ∥ x ∥ = 1 } \{\boldsymbol x:\left\|\boldsymbol x\right\|= 1\} { x:∥x∥=1} in ${\mathbb{R}}^3$ onto an ellipse in ${\mathbb{R}}^2$, shown in Figure 1. Find a unit vector $\mathbf{x}$ at which the length $\Vert A\mathbf{x}\Vert$ is maximized, and compute this maximum length.

SOLUTION
Observe that

Also, $A^{T}A$ is a symmetric matrix. So the problem now is to maximize the quadratic form ${\mathbf{x}}^T(A^{T}A)\mathbf{x}$ subject to the constraint $\Vert \mathbf{x}\Vert =1$. By Theorem 6 in Section 7.3, the maximum value is the greatest eigenvalue ${\lambda }_1$ of $A^{T}A$. Also, the maximum value is attained at a unit eigenvector of $A^{T}A$ corresponding to ${\lambda }_1$.

The eigenvalues of $A^{T}A$ are ${\lambda }_1=360,{\lambda }_2=90$, and ${\lambda }_3=0$. Corresponding unit eigenvectors are, respectively,

$A{\mathbf{v}}_1$ is a point on the ellipse in Figure 1 farthest from the origin.

The Singular Values of an $m\times n$ Matrix

Let $A$ be an $m\times n$ matrix. Then $A^{T}A$ is symmetric and can be orthogonally diagonalized. Let { v 1 , . . . , v n } \{\boldsymbol v_1,...,\boldsymbol v_n\} { v1​,...,vn​} be an orthonormal basis for ${\mathbb{R}}^n$ consisting of eigenvectors of $A^{T}A$, and let { λ 1 , . . . , λ n } \{\lambda_1,...,\lambda_n\} { λ1​,...,λn​} be the associated eigenvalues of $A^{T}A$. Then, for $1\le i\le n$,

So the eigenvalues of $A^{T}A$ are all nonnegative. By renumbering, if necessary, we may assume that the eigenvalues are arranged so that

The singular values of $A$ are the square roots of the eigenvalues of $A^{T}A$, denoted by ${\sigma }_1,...,{\sigma }_n$, and they are arranged in decreasing order. By equation (2), the singular values of $A$ are the lengths of the vectors $A{\mathbf{v}}_1,...,A{\mathbf{v}}_n$.

PROOF
For $i\ne j$ ,

Thus { A v 1 , . . . , A v n } \{A\boldsymbol v_1,...,A\boldsymbol v_n\} { Av1​,...,Avn​} is an orthogonal set. Furthermore, since the lengths of the vectors $A{\mathbf{v}}_1,...,A{\mathbf{v}}_n$ are the singular values of $A$, and since there are $r$ nonzero singular values, $A{\mathbf{v}}_i\ne 0$ if and only if $1\le i\le r$. So $A{\mathbf{v}}_1,...,A{\mathbf{v}}_r$ are linearly independent vectors, and they are in $ColA$. Finally, for any $\mathbf{y}=A\mathbf{x}$ in $ColA$, we can write $\mathbf{x}=c_1{\mathbf{v}}_1+...+c_n{\mathbf{v}}_n$, and

Thus $\mathbf{y}$ is in S p a n { A v 1 , . . . , A v r } Span\{A\boldsymbol v_1,...,A\boldsymbol v_r\} Span{ Av1​,...,Avr​}, which shows that { A v 1 , . . . , A v r } \{A\boldsymbol v_1,...,A\boldsymbol v_r\} { Av1​,...,Avr​} is an (orthogonal) basis for $ColA$. Hence $rankA=dimColA=r$.

The first singular value ${\sigma }_1$ of an $m\times n$ matrix $A$ is the maximum of $\Vert Ax\Vert$ over all unit vectors. This maximum value is attained at a unit eigenvector ${\mathbf{v}}_1$ of $A^{T}A$ corresponding to the greatest eigenvalue ${\lambda }_1$ of $A^{T}A$. The second singular value is the maximum of $\Vert Ax\Vert$ over all unit vectors orthogonal to ${\mathbf{v}}_1$.

The Singular Value Decomposition (SVD)

The decomposition of $A$ involves an $m\times n$ “diagonal” matrix $\sum$ of the form

where $D$ is an $r\times r$ diagonal matrix for some $r$ not exceeding the smaller of $m$ and $n$. (If $r$ equals $m$ or $n$ or both, some or all of the zero matrices do not appear.)

The matrices $U$ and $V$ are not uniquely determined by $A$. The columns of $U$ in such a decomposition are called left singular vectors of $A$, and the columns of $V$ are called right singular vectors of $A$.

PROOF
Let ${\lambda }_i$ and ${\mathbf{v}}_i$ be as in Theorem 9, so that { A v 1 , . . . , A v r } \{A\boldsymbol v_1,...,A\boldsymbol v_r\} { Av1​,...,Avr​} is an orthogonal basis for $ColA$. Normalize each $A{\mathbf{v}}_i$ to obtain an orthonormal basis { u 1 , . . . , u r } \{\boldsymbol u_1,...,\boldsymbol u_r\} { u1​,...,ur​}, where

and

Now extend { u 1 , . . . , u r } \{\boldsymbol u_1,...,\boldsymbol u_r\} { u1​,...,ur​} to an orthonormal basis { u 1 , . . . , u m } \{\boldsymbol u_1,...,\boldsymbol u_m\} { u1​,...,um​} of ${\mathbb{R}}^m$, and let

By construction, $U$ and $V$ are orthogonal matrices. Also, from (4),

Let $D$ be the diagonal matrix with diagonal entries ${\sigma }_1,...,{\sigma }_r$ , and let $\sum$ be as in (3) above. Then

Since $V$ is an orthogonal matrix,

The next two examples focus attention on the internal structure of a singular value decomposition. An efficient and numerically stable algorithm for this decomposition would use a different approach. See the Numerical Note at the end of the section.

EXAMPLE 3
Use the results of Examples 1 to construct a singular value decomposition of $A=\left[\begin{array}{ccc}4& 11& 14\\ 8& 7& -2\end{array}\right]$

SOLUTION
A construction can be divided into three steps.

Step 1. Find an orthogonal diagonalization of $A^{T}A$.
Step 2. Set up $V$ and $\sum$. Arrange the eigenvalues of $A^{T}A$ in decreasing order. In Example 1, the eigenvalues are already listed in decreasing order: $360,90$, and $0$. The corresponding unit eigenvectors, ${\mathbf{v}}_1,{\mathbf{v}}_2$, and ${\mathbf{v}}_3$, are the right singular vectors of $A$.

The square roots of the eigenvalues are the singular values:

Step 3. Construct $U$. When $A$ has rank $r$, the first $r$ columns of $U$ are the normalized vectors obtained from $A{\mathbf{v}}_1,...,A{\mathbf{v}}_r$ . In this example, $A$ has two nonzero singular values, so $rankA=2$. Recall that $\Vert A{\mathbf{v}}_1\Vert ={\sigma }_1$ and $\Vert A{\mathbf{v}}_2\Vert ={\sigma }_2$. Thus

Note that { u 1 , u 2 } \{\boldsymbol u_1,\boldsymbol u_2\} { u1​,u2​} is already a basis for ${\mathbb{R}}^2$. Thus no additional vectors are needed for $U$. The singular value decomposition of $A$ is

EXAMPLE 4
Find a singular value decomposition of

SOLUTION
The eigenvalues of $A^{T}A$ are 18 and 0, with corresponding unit eigenvectors

To construct $U$, first construct $A{\mathbf{v}}_1$ and $A{\mathbf{v}}_2$:

The only column found for $U$ so far is

The other columns of $U$ are found by extending the set { u 1 } \{\boldsymbol u_1\} { u1​} to an orthonormal basis for ${\mathbb{R}}^3$. In this case, we need two orthogonal unit vectors ${\mathbf{u}}_2$ and ${\mathbf{u}}_3$ that are orthogonal to ${\mathbf{u}}_1$. Each vector must satisfy ${\mathbf{u}}_1^T\mathbf{x}=0$, which is equivalent to the equation $x_1-2x_2+2x_3=0$. A basis for the solution set of this equation is

Apply the Gram–Schmidt process (with normalizations) to { w 1 , w 2 } \{\boldsymbol w_1,\boldsymbol w_2\} { w1​,w2​}, and obtain

Another way to find ${\mathbf{u}}_2$ and ${\mathbf{u}}_3$ is to realize that ${\mathbf{u}}_1$ form an orthonormal basis for $ColA$. The remaining ${\mathbf{u}}_2$ and ${\mathbf{u}}_3$ must be a basis for $(ColA{)}^{\perp }=Nul{A}^T$.

---

The next few exercises show some interesting facts.

EXERCISE
How are the singular values of $A$ and $A^T$ related?
SOLUTION
$A^T=(U\sum V^T{)}^T=V{\sum }^T{U}^T$. This is an SVD of $A^T$ because $V$ and $U$ are orthogonal matrices and ${\sum }^T$ is an $n\times m$ “diagonal” matrix. Since $\sum$ and ${\sum }^T$ have the same nonzero diagonal entries, $A$ and $A^T$ have the same nonzero singular values.
[Note: If $A$ is $2\times n$, then $A{A}^T$ is only $2\times 2$ and its eigenvalues may be easier to compute (by hand) than the eigenvalues of $A^{T}A$.]

EXERCISE 17
Show that if $A$ is square, then $|detA|$ is the product of the singular values of $A$.
SOLUTION
$|detA|=|det(U\sum V^T)|=|detU\cdot det\sum \cdot det{V}^T|=|\pm 1\cdot det\sum \cdot \pm 1|=det\sum$

EXERCISE 19
$A$ is an $m\times n$ matrix with a singular value decomposition $A=U\sum V^T$ , where $U$ is an $m\times m$ orthogonal matrix, $\sum$ is an $m\times n$ “diagonal” matrix with $r$ positive entries and no negative entries, and $V$ is an $n\times n$ orthogonal matrix. Show that the columns of $V$ are eigenvectors of $A^{T}A$, the columns of $U$ are eigenvectors of $A{A}^T$ , and the diagonal entries of $\sum$ are the singular values of $A$.
SOLUTION
[Hint: Use the SVD to compute $A^{T}A$ and $A{A}^T$ .]

EXERCISE 20
Show that if $P$ is an orthogonal $m\times m$ matrix, then $PA$ has the same singular values as $A$.

EXERCISE 22
Show that if $A$ is an $n\times n$ positive definite matrix, then an orthogonal diagonalization $A=PD{P}^T$ is a singular value decomposition of $A$.

EXERCISE 23
Let

, where the ${\mathbf{u}}_i$ and ${\mathbf{v}}_i$ are as in Theorem 10. Show that

SOLUTION

This expansion generalizes the spectral decomposition in Section 7.1.

EXERCISE 25
Let $T:{\mathbb{R}}^n\mapsto {\mathbb{R}}^m$ be a linear transformation. Describe how to find a basis $\mathcal{B}$ for ${\mathbb{R}}^n$ and a basis $\mathcal{C}$ for ${\mathbb{R}}^m$ such that the matrix for $T$ relative to $\mathcal{B}$ and $\mathcal{C}$ is an $m\times n$ “diagonal” matrix.
SOLUTION
Consider the SVD for the standard matrix of $T$, say, $A=U\sum V^T$. Let B = { v 1 , … , v n } \mathcal B = \{\boldsymbol v_1, …, \boldsymbol v_n\} B={ v1​,…,vn​} and C = { u 1 , … , u m } C = \{\boldsymbol u_1, …, \boldsymbol u_m\} C={ u1​,…,um​} be bases constructed from the columns of $V$ and $U$, respectively. Observe that, since the columns of $V$ are orthonormal, $V^T{\mathbf{v}}_j={\mathbf{e}}_j$, where ${\mathbf{e}}_j$ is the $j$th column of the $n\times n$ identity matrix. To find the matrix of $T$ relative to $\mathcal{B}$ and $\mathcal{C}$, compute

So $[T({\mathbf{v}}_j){]}_{\mathcal{C}}={\sigma }_j{\mathbf{e}}_j$. The discussion at the beginning of Section 5.4 shows that the “diagonal” matrix $\sum$ is the matrix of $T$ relative to $\mathcal{B}$ and $\mathcal{C}$.

EXERCISE
Prove that any $n\times n$ matrix $A$ admits a polar decomposition(极分解) of the form $A=PQ$, where $P$ is an $n\times n$ positive semidefinite matrix with the same rank as $A$ and where $Q$ is an $n\times n$ orthogonal matrix.
SOLUTION
[Hint: Use a singular value decomposition, $A=U\sum V^T$ , and observe that $A=(U\sum U^T)(U{V}^T)$ and $U\sum U^T$ is a symmetric matrix.]

Applications of the Singular Value Decomposition

The SVD is often used to estimate the rank of a matrix, as noted above. Several other numerical applications are described briefly below, and an application to image processing is presented in Section 7.5.

---

EXAMPLE 5 (The Condition Number (条件数))
Most numerical calculations involving an equation $A\mathbf{x}=\mathbf{b}$ are as reliable as possible when the SVD of $A$ is used. The two orthogonal matrices $U$ and $V$ do not affect lengths of vectors or angles between vectors. Any possible instabilities in numerical calculations are identified in $\sum$. If the singular values of $A$ are extremely large or small, roundoff errors are almost inevitable, but an error analysis is aided by knowing the entries in $\sum$ and $V$ .
If $A$ is an invertible $n\times n$ matrix, then the ratio ${\sigma }_1={\sigma }_n$ of the largest and smallest singular values gives the condition number of $A$. Exercises 41–43 in Section 2.3 showed how the condition number affects the sensitivity of a solution of $A\mathbf{x}=\mathbf{b}$ to changes (or errors) in the entries of A. (Actually, a “condition number” of $A$ can be computed in several ways, but the definition given here is widely used for studying $A\mathbf{x}=\mathbf{b}$.)

---

EXAMPLE 6 (Bases for Fundamental Subspaces)
Given an SVD for an $m\times n$ matrix $A$, let ${\mathbf{u}}_1,...,{\mathbf{u}}_m$ be the left singular vectors,${\mathbf{v}}_1,...,{\mathbf{v}}_n$ the right singular vectors, and ${\sigma }_1,...,{\sigma }_n$ the singular values, and let $r$ be the rank of $A$. By Theorem 9,
{ u 1 , . . . , u r }      ( 5 ) \{\boldsymbol u_1,...,\boldsymbol u_r\}\ \ \ \ (5) { u1​,...,ur​}    (5)

is an orthonormal basis for $ColA$.

Recall that $(ColA{)}^{\perp }=Nul{A}^T$ . Hence
{ u r + 1 , . . . , u m }      ( 6 ) \{\boldsymbol u_{r+1},...,\boldsymbol u_m\}\ \ \ \ (6) { ur+1​,...,um​}    (6)

is an orthonormal basis for $Nul{A}^T$ .

Since $\Vert A{\mathbf{v}}_i\Vert ={\sigma }_i$ for $1\le i\le n$, and ${\sigma }_i$ is 0 if and only if $i>r$, the vectors ${\mathbf{v}}_{r+1},...,{\mathbf{v}}_n$ span a subspace of $NulA$ of dimension $n-r$. By the Rank Theorem, $dimNulA=n-rankA=n-r$. It follows that

{ v r + 1 , . . . , v n }      ( 7 ) \{\boldsymbol v_{r+1},...,\boldsymbol v_n\}\ \ \ \ (7) { vr+1​,...,vn​}    (7)

is an orthonormal basis for $NulA$.

$(NulA{)}^{\perp }=Col{A}^T=RowA$. Hence, from $(7)$,

{ v 1 , . . . , v r }      ( 8 ) \{\boldsymbol v_1,...,\boldsymbol v_r\}\ \ \ \ (8) { v1​,...,vr​}    (8)

is an orthonormal basis for $RowA$.

Figure 4 summarizes $(5)–(8)$, but shows the orthogonal basis { σ 1 u 1 , . . . , σ r u r } \{\sigma_1\boldsymbol u_1,...,\sigma_r\boldsymbol u_r\} { σ1​u1​,...,σr​ur​} for $ColA$ instead of the normalized basis, to remind you that $A{\mathbf{v}}_i={\sigma }_i{\mathbf{u}}_i$ for $1\le i\le r$.

The four fundamental subspaces and the concept of singular values provide the final statements of the Invertible Matrix Theorem.

---

EXAMPLE 7 (Reduced SVD and the Pseudoinverse of $A$ (奇异值分解的简化和 $A$ 的伪逆))
When $\sum$ contains rows or columns of zeros, a more compact decomposition of $A$ is possible. Using the notation established above, let $r=rankA$, and partition $U$ and $V$ into submatrices whose first blocks contain $r$ columns:

Then $U_r$ is $m\times r$ and $V_r$ is $n\times r$. (To simplify notation, we consider $U_{m-r}$ or $V_{n-r}$ even though one of them may have no columns.) Then partitioned matrix multiplication shows that

This factorization of $A$ is called a reduced singular value decomposition of $A$. Since the diagonal entries in $D$ are nonzero, $D$ is invertible. The following matrix is called the pseudoinverse (伪逆) (also, the Moore–Penrose inverse (穆尔-彭罗斯逆)) of $A$:

The next Supplementary exercises explore some of the properties of the reduced singular value decomposition and the pseudoinverse.

Supplementary EXERCISE 12
Verify the properties of $A^{+}$:
a. For each $\mathbf{y}$ in ${\mathbb{R}}^m$, $A{A}^{+}\mathbf{y}$ is the orthogonal projection of $\mathbf{y}$ onto $ColA$.
b. For each $\mathbf{x}$ in ${\mathbb{R}}^n$, $A^{+}A\mathbf{x}$ is the orthogonal projection of $\mathbf{x}$ onto $RowA$.
c. $A{A}^{+}A=A$ and $A^{+}A{A}^{+}=A^{+}$.

Supplementary EXERCISE 13
Suppose the equation $A\mathbf{x}=\mathbf{b}$ is consistent, and let ${\mathbf{x}}^{+}=A^{+}\mathbf{b}$. By Exercise 23 in Section 6.3, there is exactly one vector $\mathbf{p}$ in $RowA$ such that $A\mathbf{p}=\mathbf{b}$. The following steps prove that ${\mathbf{x}}^{+}=\mathbf{p}$ and ${\mathbf{x}}^{+}$ is the minimum length solution of $A\mathbf{x}=\mathbf{b}$.
a. Show that ${\mathbf{x}}^{+}$ is in $RowA$.
b. Show that ${\mathbf{x}}^{+}$ is a solution of $A\mathbf{x}=\mathbf{b}$.
c. Show that if $\mathbf{u}$ is any solution of $A\mathbf{x}=\mathbf{b}$, then $\Vert {\mathbf{x}}^{+}\Vert \le \Vert \mathbf{u}\Vert$, with equality only if $\mathbf{u}={\mathbf{x}}^{+}$.
SOLUTION
a. ${\mathbf{x}}^{+}=V_r{D}^{-1}{U}_r^T$. Since the columns of $V_r$ form an orthonormal basis for $RowA$, ${\mathbf{x}}^{+}$ is a linear combination of the $RowA$'s orthonormal basis. Thus ${\mathbf{x}}^{+}$ is in $RowA$.
b. $A{\mathbf{x}}^{+}=A{A}^{+}\mathbf{b}=A{A}^{+}A\mathbf{x}=A\mathbf{x}=\mathbf{b}$
c. ${\mathbf{x}}^{+}$ is the orthogonal projection of $\mathbf{u}$ onto $RowA$. …

Supplementary EXERCISE 14
Given any $\mathbf{b}$ in ${\mathbb{R}}^m$, adapt Exercise 13 to show that $A^{+}\mathbf{b}$ is the least-squares solution of minimum length.
SOLUTION
[Hint: Consider the equation $A\mathbf{x}=\hat{\mathbf{b}}$, where $\hat{\mathbf{b}}$ is the orthogonal projection of $\mathbf{b}$ onto $ColA$.]

---

EXAMPLE 8 (Least-Squares Solution)
Given the equation $A\mathbf{x}=\mathbf{b}$, use the pseudoinverse of $A$ to define

Then,

$U_r{U}_r^T\mathbf{b}$ is the orthogonal projection $\hat{\mathbf{b}}$ of $\mathbf{b}$ onto $ColA$. Thus $\hat{\mathbf{x}}$ is a least-squares solution of $A\mathbf{x}=\mathbf{b}$. In fact, this $\hat{\mathbf{x}}$ has the smallest length among all least-squares solutions of $A\mathbf{x}=\mathbf{b}$. See Supplementary Exercise 14.

---