6.4 The Gram–Schmidt process (格拉姆-施密特算法)

本文为《Linear algebra and its applications》的读书笔记

The Gram–Schmidt process

The Gram–Schmidt process is a simple algorithm for producing an orthogonal or orthonormal basis for any nonzero subspace of ${\mathbb{R}}^n$.

EXAMPLE 1
Let W = S p a n { x 1 , x 2 } W = Span\{\boldsymbol x_1, \boldsymbol x_2\} W=Span{ x1​,x2​}, where ${\mathbf{x}}_1=\left[\begin{array}{c}3\\ 6\\ 0\end{array}\right]$ and ${\mathbf{x}}_2=\left[\begin{array}{c}1\\ 2\\ 3\end{array}\right]$. Construct an orthogonal basis { v 1 , v 2 } \{\boldsymbol v_1, \boldsymbol v_2\} { v1​,v2​} for $W$.
SOLUTION

The component of ${\mathbf{x}}_2$ orthogonal to ${\mathbf{x}}_1$ is ${\mathbf{x}}_2-\mathbf{p}$, which is in $W$. Let ${\mathbf{v}}_1={\mathbf{x}}_1$ and

The next example fully illustrates the Gram–Schmidt process. Study it carefully.

EXAMPLE 2 Let ${\mathbf{x}}_1=\left[\begin{array}{c}1\\ 1\\ 1\\ 1\end{array}\right]$, ${\mathbf{x}}_2=\left[\begin{array}{c}0\\ 1\\ 1\\ 1\end{array}\right]$, and ${\mathbf{x}}_3=\left[\begin{array}{c}0\\ 0\\ 1\\ 1\end{array}\right]$. Then { x 1 , x 2 , x 3 } \{\boldsymbol x_1, \boldsymbol x_2,\boldsymbol x_3\} { x1​,x2​,x3​} is clearly linearly independent and thus is a basis for a subspace $W$ of ${\mathbb{R}}^4$. Construct an orthogonal basis for $W$ .
SOLUTION
$Step1.$ Let ${\mathbf{v}}_1={\mathbf{x}}_1$ and W 1 = S p a n { x 1 } = S p a n { v 1 } W_1 = Span\{\boldsymbol x_1\}= Span\{\boldsymbol v_1\} W1​=Span{ x1​}=Span{ v1​}.

$Step2.$ Let ${\mathbf{v}}_2$ be the vector produced by subtracting from ${\mathbf{x}}_2$ its projection onto the subspace $W_1$. That is, let

Then { v 1 , v 2 } \{\boldsymbol v_1,\boldsymbol v_2\} { v1​,v2​} is an orthogonal basis for the subspace $W_2$ spanned by ${\mathbf{x}}_1$ and ${\mathbf{x}}_2$.

$Step{2}^{\prime }(optional).$ If appropriate, scale ${\mathbf{v}}_2$ to simplify later computations. Since ${\mathbf{v}}_2$ has fractional entries, it is convenient to scale it by a factor of 4:

$Step3.$ Let ${\mathbf{v}}_3$ be the vector produced by subtracting from ${\mathbf{x}}_3$ its projection onto the subspace $W_2$. Use the orthogonal basis { v 1 , v 2 ′ } \{\boldsymbol v_1,\boldsymbol v_2'\} { v1​,v2′​} to compute this projection onto $W_2$:

The proof of the next theorem shows that this strategy really works. Scaling of vectors is not mentioned because that is used only to simplify hand calculations.

Theorem 11 shows that any nonzero subspace $W$ of ${\mathbb{R}}^n$ has an orthogonal basis.

PROOF
For $1\le k\le p$, let W k = S p a n { x 1 , . . . , x k } W_k = Span \{\boldsymbol x_1,..., \boldsymbol x_k\} Wk​=Span{ x1​,...,xk​}.

Set ${\mathbf{v}}_1={\mathbf{x}}_1$, so that S p a n { v 1 } = S p a n { x 1 } Span \{v_1\}= Span \{x_1\} Span{ v1​}=Span{ x1​}.

Suppose, for some $k<p$, we have constructed ${\mathbf{v}}_1,...,{\mathbf{v}}_k$ so that { v 1 , . . . , v k } \{\boldsymbol v_1,..., \boldsymbol v_k\} { v1​,...,vk​} an orthogonal basis for $W_k$. Define
$${\mathbf{v}}_{k+1}={\mathbf{x}}_{k+1}-pro{j}_{W_k}{\mathbf{x}}_{k+1}(2)$$

By the Orthogonal Decomposition Theorem, ${\mathbf{v}}_{k+1}$ is orthogonal to $W_k$.
Note that $pro{j}_{W_k}{\mathbf{x}}_{k+1}$ is in $W_k$ and hence also in $W_{k+1}$. Since ${\mathbf{x}}_{k+1}$ is in $W_{k+1}$, so is ${\mathbf{v}}_{k+1}$.
Furthermore, ${\mathbf{v}}_{k+1}\ne 0$ because ${\mathbf{x}}_{k+1}$ is not in $W_k$.
Hence { v 1 , . . . , v k + 1 } \{\boldsymbol v_1,..., \boldsymbol v_{k+1}\} { v1​,...,vk+1​} is an orthogonal set of nonzero vectors in $W_{k+1}$. This set is an orthogonal basis for $W_{k+1}$. Hence W k + 1 = S p a n { v 1 , . . . , v k + 1 } W_{k+1}= Span \{\boldsymbol v_1,..., \boldsymbol v_{k+1}\} Wk+1​=Span{ v1​,...,vk+1​}. When $k+1=p$, the process stops.

Orthonormal Bases

An orthonormal basis is constructed easily from an orthogonal basis { v 1 , . . . , v p } \{\boldsymbol v_1,..., \boldsymbol v_p\} { v1​,...,vp​}: simply normalize (i.e., “scale”) all the ${\mathbf{v}}_k$.

When working problems by hand, this is easier than normalizing each ${\mathbf{v}}_k$ as soon as it is found (because it avoids unnecessary writing of square roots).

QR Factorization of Matrices

If an $m\times n$ matrix $A$ has linearly independent columns { x 1 , . . . , x n } \{\boldsymbol x_1,..., \boldsymbol x_{n}\} { x1​,...,xn​}, then applying the Gram–Schmidt process (with normalizations) to { x 1 , . . . , x n } \{\boldsymbol x_1,..., \boldsymbol x_{n}\} { x1​,...,xn​} amounts to $factoring$ $A$.

This factorization is widely used in computer algorithms for various computations, such as solving equations (discussed in Section 6.5) and finding eigenvalues (mentioned in the exercises for Section 5.2).

PROOF
The columns of $A$ form a basis { x 1 , . . . , x n } \{\boldsymbol x_1,..., \boldsymbol x_{n}\} { x1​,...,xn​} for $ColA$. Construct an orthonormal basis { u 1 , . . . , u n } \{\boldsymbol u_1,..., \boldsymbol u_{n}\} { u1​,...,un​} for $W=ColA$. This basis may be constructed by the Gram–Schmidt process or some other means. Let

For $k=1,...,n$, ${\mathbf{x}}_k$ is in S p a n { x 1 , . . . , x k } = S p a n { u 1 , . . . , u k } Span \{\boldsymbol x_1,...,\boldsymbol x_k\}= Span\{\boldsymbol u_1,...,\boldsymbol u_k\} Span{ x1​,...,xk​}=Span{ u1​,...,uk​}. So there are constants, $r_{1k},...,r_{kk}$, such that

We may assume that $r_{kk}\ge 0$. (If $r_{kk}<0$, multiply both $r_{kk}$ and ${\mathbf{u}}_k$ by $-1$.) This shows that ${\mathbf{x}}_k$ is a linear combination of the columns of $Q$ using as weights the entries in the vector

That is, ${\mathbf{x}}_k=Q{\mathbf{r}}_k$ for $k=1,...,n$. Let $R=[{\mathbf{r}}_1...{\mathbf{r}}_n]$. Then

The fact that $R$ is invertible follows easily from the fact that the columns of $A$ are linearly independent. Since $R$ is clearly upper triangular, its nonnegative diagonal entries must be positive.

EXERCISES
Suppose $A=QR$, where $Q$ is $m\times n$ and $R$ is $n\times n$. Show that if the columns of $A$ are linearly independent, then $R$ must be invertible.
SOLUTION
[Hint: Study the equation $R\mathbf{x}=0$ and use the fact that $A=QR$.]

---

$Q$ can be constructed by Gram–Schmidt process and $R$ can be calculated by $R=Q^{T}A$

${}^1$ See $Fundamentals$ $of$ $Matrix$ $Computations$, by $David$ $S.$ $Watkins$ (New York: John Wiley & Sons, 1991), pp. 167–180.

EXERCISES 23
Suppose $A=QR$ is a $QR$ factorization of an $m\times n$ matrix $A$ (with linearly independent columns). Partition $A$ as $[A_1{A}_2]$, where $A_1$ has $p$ columns. Show how to obtain a QR factorization of $A_1$ .
SOLUTION
$$\begin{gathered} \because A=QR \\ \therefore \left[\begin{array}{cc}{A}_{1_{m\times p}}& A_{2_{m\times (n-p)}}\end{array}\right]=\left[\begin{array}{cc}{Q}_{1_{m\times p}}& Q_{2_{m\times (n-p)}}\end{array}\right]\left[\begin{array}{cc}{R}_{11_{p\times p}}& R_{12_{p\times (n-p)}}\\ 0_{(n-p)\times p}& R_{22_{(n-p)\times (n-p)}}\end{array}\right] \\ \therefore A_{1_{m\times p}}=Q_{1_{m\times p}}{R}_{11_{p\times p}} \end{gathered}$$