7.3 Constrained optimization (条件优化)

本文为《Linear algebra and its applications》的读书笔记

Constrained optimization

Engineers, economists, scientists, and mathematicians often need to find the maximum or minimum value of a quadratic form $Q(\mathbf{x})$ for $\mathbf{x}$ in some specified set. Typically, the problem can be arranged so that $\mathbf{x}$ varies over the set of unit vectors. This constrained optimization problem has an interesting and elegant solution. Example 6 below and the discussion in Section 7.5 will illustrate how such problems arise in practice.

The requirement that a vector $\mathbf{x}$ in ${\mathbb{R}}^n$ be a unit vector can be stated in several equivalent ways:

and

The expanded version (1) of ${\mathbf{x}}^T\mathbf{x}=1$ is commonly used in applications.

When a quadratic form $Q$ has no cross-product terms, it is easy to find the maximum and minimum of $Q(\mathbf{x})$ for ${\mathbf{x}}^T\mathbf{x}=1$.

EXAMPLE 1
Find the maximum and minimum values of $Q(\mathbf{x})=9x_1^2+4x_2^2+3x_3^2$ subject to the constraint ${\mathbf{x}}^T\mathbf{x}=1$.
SOLUTION

Furthermore,$Q(\mathbf{x})=9$ when $\mathbf{x}=(1,0,0)$. Thus 9 is the maximum value of $Q(\mathbf{x})$ for ${\mathbf{x}}^T\mathbf{x}=1$.

Also,$Q(\mathbf{x})=9$ when $\mathbf{x}=(0,0,1)$. Thus 3 is the minimum value of $Q(\mathbf{x})$ for ${\mathbf{x}}^T\mathbf{x}=1$.

It is easy to see in Example 1 that the matrix of the quadratic form $Q$ has eigenvalues $9,4$, and $3$ and that the greatest and least eigenvalues equal, respectively, the (constrained) maximum and minimum of $Q(\mathbf{x})$. The same holds true for any quadratic form, as we shall see.

EXAMPLE 2
Let $A=\left[\begin{array}{cc}3& 0\\ 0& 7\end{array}\right]$, and let $Q(\mathbf{x})={\mathbf{x}}^{T}A\mathbf{x}$ for $\mathbf{x}$ in ${\mathbb{R}}^2$. Figure 1 displays the graph of $Q$. Figure 2 shows only the portion of the graph inside a cylinder; the intersection of the cylinder with the surface is the set of points $(x_1,x_2,z)$ such that $z=Q(x_1,x_2)$ and $x_1^2+x_2^2=1$.

The two highest points on the curve are $7$ units above the $x_1{x}_2$-plane, occurring where $x_1=0$ and $x_2=\pm 1$. These points correspond to the eigenvalue $7$ of $A$ and the eigenvectors $\mathbf{x}=(0,1)$ and $-\mathbf{x}=(0,-1)$. Similarly, the two lowest points on the curve are $3$ units above the $x_1{x}_2$-plane. They correspond to the eigenvalue $3$ and the eigenvectors $\mathbf{x}=(1,0)$ and $-\mathbf{x}=(-1,0)$.

The set of all possible values of ${\mathbf{x}}^{T}A\mathbf{x}$, for $\Vert x\Vert =1$, is the closed interval $3\le t\le 7$.

It can be shown that for any symmetric matrix $A$, the set of all possible values of ${\mathbf{x}}^{T}A\mathbf{x}$, for $\Vert x\Vert =1$, is a closed interval on the real axis. (See Exercise 13.) Denote the left and right endpoints of this interval by $m$ and $M$, respectively. That is, let

---

EXERCISE 13
Let $A$ be an $n\times n$ symmetric matrix, let $M$ and $m$ denote the maximum and minimum values of the quadratic form ${\mathbf{x}}^{T}A\mathbf{x}$, where ${\mathbf{x}}^T\mathbf{x}=1$; and denote corresponding unit eigenvectors by ${\mathbf{u}}_1$ and ${\mathbf{u}}_n$.

The following calculations show that given any number $t$ between $M$ and $m$, there is a unit vector $\mathbf{x}$ such that $t={\mathbf{x}}^{T}A\mathbf{x}$.

If $m=M$ and $\mathbf{x}$ is the unit eigenvector ${\mathbf{u}}_1$, then ${\mathbf{x}}^{T}A\mathbf{x}={\mathbf{u}}_1^{T}A{\mathbf{u}}_1={\mathbf{u}}_1^T(m{\mathbf{u}}_1)=m$.

Otherwise, $0\le t-m\le M-m$ and $0\le (t-m)/(M-m)\le 1$. So set $\alpha =(t-m)/(M-m)$. Thus $t=(1-\alpha )m+\alpha M$.

For such $\alpha$, let $\mathbf{x}=\sqrt{1-\alpha }{\mathbf{u}}_n+\sqrt{\alpha }{\mathbf{u}}_1$. It can be shown that ${\mathbf{x}}^{T}A\mathbf{x}=t$ and ${\mathbf{x}}^T\mathbf{x}=1$.

---

PROOF
Orthogonally diagonalize $A$ as $PD{P}^{-1}$. We know that

Also,

Thus ${\mathbf{x}}^{T}A\mathbf{x}$ and ${\mathbf{y}}^{T}D\mathbf{y}$ assume the same set of values as $\mathbf{x}$ and $\mathbf{y}$ range over the set of all unit vectors. (Suppose that ${\lambda }_1\ge {\lambda }_2\ge ...\ge {\lambda }_n$)

$${\mathbf{x}}^{T}A\mathbf{x}={\mathbf{y}}^{T}D\mathbf{y}=\sum _{i=1}^{i=n}{\lambda }_i{y}_i^2(\ast )$$$$\begin{gathered} (\ast )\le {\lambda }_1\sum _{i=1}^{i=n}{y}_i^2 \\ (\ast )hasthemaximumvaluewhen\mathbf{y}={\mathbf{e}}_1,\mathbf{x}=P{\mathbf{e}}_1={\mathbf{u}}_1 \\ (\ast )\ge {\lambda }_n\sum _{i=1}^{i=n}{y}_i^2 \\ (\ast )hastheminimumvaluewhen\mathbf{y}={\mathbf{e}}_n,\mathbf{x}=P{\mathbf{e}}_n={\mathbf{u}}_n \end{gathered}$$

In Theorem 7 and in later applications, the values of ${\mathbf{x}}^{T}A\mathbf{x}$ are computed with additional constraints on the unit vector $\mathbf{x}$.

The proof is similar to Theorem 6.

The next theorem generalizes Theorem 7 and, together with Theorem 6, gives a useful characterization of all the eigenvalues of $A$.

EXAMPLE 6
During the next year, a county government is planning to repair $x$ hundred miles of public roads and bridges and to improve $y$ hundred acres of parks and recreation areas. The county must decide how to allocate its resources (funds, equipment, labor, etc.) between these two projects. If it is more cost effective to work simultaneously on both projects rather than on only one, then $x$ and $y$ might satisfy a constraint such as

See Figure 3. Each point $(x,y)$ in the shaded feasible set represents a possible public works schedule for the year. The points on the constraint curve, $4x^2+9y^2=36$, use the maximum amounts of resources available.

In choosing its public works schedule, the county wants to consider the opinions of the county residents. To measure the value, or $utility(效用)$, that the residents would assign to the various work schedules $(x,y)$, economists sometimes use a function such as

The set of points $(x,y)$ at which $q(x,y)$ is a constant is called an $indifference$ $curve$(无差别曲线). Find the public works schedule that maximizes the utility function $q$.

SOLUTION
The constraint equation does not describe a set of unit vectors, but a change of variable can fix that problem. Rewrite the constraint in the form

and define

Then the constraint equation becomes

and the utility function becomes $q(3x_1,2x_2)=(3x_1)(2x_2)=6x_1{x}_2$. Let $\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]$. Then the problem is to maximize $Q(\mathbf{x})=6x_1{x}_2$ subject to ${\mathbf{x}}^T\mathbf{x}=1$. Note that $Q(\mathbf{x})={\mathbf{x}}^{T}A\mathbf{x}$, where

The eigenvalues of $A$ are $\pm 3$, with eigenvectors

Thus the maximum value of $Q(\mathbf{x})$ is 3, attained when $x_1=1/\sqrt{2}$ and $x_2=1/\sqrt{2}$.

In terms of the original variables, the optimum public works schedule is $x=3x_1=3/\sqrt{2}\approx 2.1$ hundred miles of roads and bridges and $y=2x_2=\sqrt{2}\approx 1.4$ hundred acres of parks and recreational areas. The optimum public works schedule is the point where the constraint curve and the indifference curve just meet.