[Optimization Method Study Notes] Chapter 3: Constrained Optimization Method

1. Constrained optimization problem

1.1 General form of constrained optimization problem

The general form of the constrained optimization problem is$$\begin{array}{ccc}\min & f(\mathbf{x})& \\ s.t.& h_i(\mathbf{x})=0& i=1,2,\cdots ,l\\ & h_j\left(x\right)\le 0& j=l+1,l+2,\cdots ,m\end{array}$$称$f\left(x\right)$为Object function number, $h_i\left(x\right)=0$为Equation promise, $h_j\left(x\right)\le 0$为不等式约束。
称集合 Ω = { x ∣ h i ( x ) = 0 , h j ( x ) ≤ 0 , i = 1 , 2 , ⋯   , l , j = l + 1 , l + 2 , ⋯   , m } \varOmega = \left \lbrace \boldsymbol{x} | h_i(\boldsymbol{x}) = 0, h_j(\boldsymbol{x}) \le 0, i = 1, 2, \cdots, l, j = l+1, l+2, \cdots, m \right \rbrace Oh={ x∣hi​(x)=0,hj​(x)≤0,i=1,2,⋯,l,j=l+1,l+2,⋯,m}为Allowable range.

1.2 Feasible direction and feasible descent direction

设$d$Non-zero direction quantity, $\mathbf{x}\in \Omega$, 若$\exists k>0$, 使得$\forall \alpha \in (0,k)$, 都有$x+\alpha \mathbf{d}\in \Omega$, symmetric direction quantity$d$为point$x$ feasible direction, if it still satisfies$f(\mathbf{x}+\alpha \mathbf{d})<f\left(x\right)$, 则名$d$为point x \boldsymbol{x} feasible descent direction at ). Possible directions for improvement (or $x$

1.3 Effective indicator set

对于Point$x\in \Omega$, 称集合 A ( x ) = { i ∣ h i ( x ) = 0 } A(\boldsymbol{x}) = \left \lbrace i | h_i(\boldsymbol{x}) = 0 \right \rbrace A(x)={ i∣hi​(x)=0}为point$x$'seffective indicator set, intuitively speaking, the effective indicator set is all etc. The set of the subscripts of expression constraints and the subscripts of all inequality constraints with the equal sign.

2. KKT conditions

Suppose the general form of constrained optimization problem is at point $x$处满 Foot：Directions组$\nabla h_i(\mathbf{x})$, $i\in A\left(x\right)$Wirelessness, problem exists Point$x$处的拉格郎日function为$$L(\mathbf{x},\mathbf{\lambda })=f(\mathbf{x})+\sum _{i=1}^m{l}_i{h}_i\left(x\right)$$Where is the problem$\mathbf{x}$处的KKT条件为$$\{\begin{array}{l}{\nabla }_{\mathbf{x}}L(x,\lambda )=0\\ h_i(\mathbf{x})=0,i=1,2,\cdots ,l\\ h_j\left(x\right)\le 0,j=l+1,l+2,\cdots ,m\\ l_j{h}_j\left(x\right)=0,j=l+1,l+2,\cdots ,m\\ l_j\ge 0,j=l+1,l+2,\cdots ,m\end{array}$$若点$x$Complete KKT condition, other name$x$为KKT point, mutually$(\mathbf{x},\lambda )$Name为KKT对.

[Example 1] Find all KKT points of the following problems$$\begin{array}{cc}\min & x_1{x}_2\\ s.t.& x_1^2+x_2^2=1\end{array}$$[Solution] Construct Lagrangian function$$L(x_1,x_2,\lambda )=x_1{x}_2+\lambda x_1^2+\lambda x_2^2-\lambda$$KKT条件为$$\{\begin{array}{l}{x}_2+\lambda x_1=0\\ x_1+\lambda x_2=0\\ x_1^2+x_2^2=1\end{array}$$x 1 = x 2 = ± 2 2 , λ = − 1 2 x_1 = x_2 = \pm \dfrac{\sqrt2}{2}, \lambda = -\dfrac{1}{2}$$x_1=x_2=\pm \frac{\sqrt{2}}{2},l=-\frac{1}{2}$$或$$x_1=-x_2=\pm \frac{\sqrt{2}}{2},l=\frac{1}{2}$$So the KKT point is${\left(\frac{\sqrt{2}}{2},\frac{\sqrt{2}}{2}\right)}^{\mathrm{T}}$, ${\left(-\frac{\sqrt{2}}{2},\frac{\sqrt{2}}{2}\right)}^{\mathrm{T}}$, ${\left(\frac{\sqrt{2}}{2},-\frac{\sqrt{2}}{2}\right)}^{\mathrm{T}}$, ${\left(-\frac{\sqrt{2}}{2},-\frac{\sqrt{2}}{2}\right)}^{\mathrm{T}}$。

[Example 2] Judgment point${\mathbf{x}}_0=(1,3{)}^{Is\; T}$ the KKT point of the following problem$$\begin{array}{cc}\min & 4x_1-3x_2\\ s.t.& x_1+x_2\le 4\\ & x_2+7\ge 0\\ & (x_1-3{)}^2\le 1+x_2\end{array}$$[Answer] Point$x_0$The set of active indicators at is A ( x 0 ) = { 1 , 3 } A(\boldsymbol{x}_0) = \lbrace 1, 3 \rbrace < /span>A(x0​)={ 1,3}, because$l_2=0$, construct Lagrangian function$$L(x_1,x_2,l_1,l_3)=4x_1-3x_2+l_1(x_1+x_2-4)+l_3\left[(x_1-3{)}^2-x_2-1\right]$$KKT EXPORTS$$\{\begin{array}{l}4+l_1+2{\lambda }_3{x}_1-6{\lambda }_3=0\\ -3+l_1-l_3=0\\ l_1\ge 0,l_3\ge 0\end{array}$$General$x_1=1$sum$x_2=3$Therefore, the equation$$\{\begin{array}{l}{l}_1-4{\lambda }_3=-4\\ l_1-l_3=3\\ l_1\ge 0,l_3\ge 0\end{array}$$The above equation has a solution:$l_1=\frac{16}{3}\ge 0$, $l_3=\frac{7}{3}\ge 0$, reason$x_0$It is the KKT point.

3. Secondary planning

3.1 General form of quadratic programming

The objective function is called a quadratic function and the constraints are linear constraints. The constrained optimization problem isquadratic programming. The general form of quadratic programming The form is$$\begin{array}{ccc}\min & \frac{1}{2}{\mathbf{x}}^{\mathrm{T}}\mathbf{G}\mathbf{x}+{\mathbf{c}}^{\mathrm{T}}\mathbf{x}& \\ s.t.& {\mathbf{a}}_i^{\mathrm{T}}x=b_i& i=1,2,\cdots ,l\\ & a_j^{T}x\le b_j& j=l+1,l+2,\cdots ,m\end{array}$$

3.2 Equality constrained quadratic programming

If the quadratic programming problem does not contain inequality constraints, the problem degenerates into$$\begin{array}{cc}\min & \frac{1}{2}{x}^{T}Gx+c^{T}x\\ s.t.& \mathbf{A}\mathbf{x}=\mathbf{b}\end{array}$$If matrix$G$semidefinite, and$All\; rows\; of\; A$ are linearly independent, then the KKT point of the problem is equivalent to the optimal solution of the problem. Just need to solve the system of linear equations$$\left[\begin{array}{cc}\mathbf{G}& {\mathbf{A}}^{\mathrm{T}}\\ \mathbf{A}& \mathbf{O}\end{array}\right]\left[\begin{array}{c}x\\ \mathbf{\lambda }\end{array}\right]=\left[\begin{array}{c}-\mathbf{c}\\ b\end{array}\right]$$ can get the optimal solution to the problem.

[Example 3] Solve quadratic programming problem$$\begin{array}{cc}\min & x_1^2+x_2^2+x_3^2-x_1{x}_2-x_2{x}_3+2x_1-x_2\\ s.t.& 3x_1-x_2-x_3=0\\ & 2x_1-x_2-x_3=0\end{array}$$[Solution] Transform the problem into matrix form$$\begin{array}{cc}\min & \frac{1}{2}[x_1,x_2,x_3]\left[\begin{array}{ccc}2& -1& 0\\ -1& 2& -1\\ 0& -1& 2\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]+[2,-1,0]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]\\ & \\ s.t.& \left[\begin{array}{ccc}3& -1& -1\\ 2& -1& -1\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right]\end{array}$$Where matrix$\left[\begin{array}{ccc}2& -1& 0\\ -1& 2& -1\\ 0& -1& 2\end{array}\right]$In other words, the operating system$$\left[\begin{array}{ccccc}2& -1& 0& 3& 2\\ -1& 2& -1& -1& -1\\ 0& -1& 2& -1& -1\\ 3& -1& -1& 0& 0\\ 2& -1& -1& 0& 0\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\\ l_1\\ l_2\end{array}\right]=\left[\begin{array}{c}-2\\ 1\\ 0\\ 0\\ 0\end{array}\right]$$得$[x_1,x_2,x_3,l_1,l_2]=\left[0,\frac{1}{6},\frac{1}{6},-\frac{5}{6},\frac{1}{3}\right]$, so the optimal solution is$[x_1,x_2,x_3]=\left[0,\frac{1}{6},\frac{1}{6}\right]$, 上优值为$-\frac{5}{36}$。

3.3 Effective indicator set method

For the general form of quadratic programming problem, if$G$ is a positive definite matrix, then the following algorithm can obtain the optimal solution to the problem:

1. Give the initial feasible point of the problem$x$
2. Initialized bottom set$I\leftarrow A(\mathbf{x})$
3. $\mathbf{while}\mathrm{True}\mathbf{do}$
4. \qquad Solve the following quadratic programming problem with only equality constraints to get $\mathbf{d}$ 和 $\mathbf{\lambda }$$$\begin{array}{cc}\underset{\mathbf{d}}{\min }& \frac{1}{2}{d}^{\mathrm{T}}\mathbf{G}\mathbf{d}+(\mathbf{G}\mathbf{x}+\mathbf{c}{)}^{\mathrm{T}}\mathbf{d}\\ s.t.& {\mathbf{a}}_i^{\mathrm{T}}d=0,i\in I\end{array}$$\qquad
5. $\mathbf{if}d=0do$
6. $ifl\ge 0do$
7. $\mathbf{return}x$
8. $\mathbf{else}$
9. I ← I ∖ { arg min ⁡ λ i } \qquad \qquad \qquad I \gets I \setminus \left \lbrace \argmin \lambda_i \right \rbrace I←I∖{ argminli​}
10. $\mathbf{end}$
11. $\mathbf{else}$
12. $a\leftarrow \underset{i\notin I}{\min }\left\{\frac{b_i-a_i^T\mathbf{x}}{{\mathbf{a}}_i^{\mathrm{T}}\mathbf{d}}\mid a_i^{T}d>0\right\}$
13. $\mathbf{if}a<1\mathbf{do}$
14. $i\leftarrow \underset{i\notin I}{\mathrm{argmin}}\left\{\frac{b_i-a_i^{T}x}{a_i^{T}d}|a_i^{T}d>0\right\}$
15. I ←   I ∪ { i } \qquad \qquad \qquad I \gets \ I \cup \lbrace i \rbrace I← I∪{ i}
16. $else$
17. $a\leftarrow 1$
18. $end$
19. $\mathbf{x}\leftarrow x+\alpha \mathbf{d}$
20. $end$
21. $end$

The above algorithm is calledactive indicator set method.

[Example 4] Solving quadratic programming problem$$\begin{array}{cc}\min & (x_1-1{)}^2+(x_2-2{)}^2\\ s.t.& x_1+x_2\le 1\\ & x_1,x_2\ge 0\end{array}$$[Solution] The coefficients corresponding to this problem are$$\mathbf{G}=\left[\begin{array}{cc}2& 0\\ 0& 2\end{array}\right],\mathbf{c}=\left[\begin{array}{c}-2\\ -4\end{array}\right],a_1=\left[\begin{array}{c}1\\ 1\end{array}\right],a_2=\left[\begin{array}{c}-1\\ 0\end{array}\right],a_3=\left[\begin{array}{c}0\\ -1\end{array}\right],b=\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right]$$显然, $x_0=[0,0{]}^T$is possible, initialization I 0 = A ( 0 , 0 ) = { 2 , 3 } I_0 = A(0, 0) = \lbrace 2, 3 \rbrace I0​=A(0,0)={ 2,3}

th $1$ iterations:
Solve quadratic programming problem$$\begin{array}{cc}\min & d_1^2+d_2^2-2d_1-4d_2\\ s.t.& -d_1=0\\ & -d_2=0\end{array}$$得$d=[0,0{]}^{\mathrm{T}}$, $l=[0,-2,-4{]}^{\mathrm{T}}\le 0$, $\mathrm{argmin}{l}_i=3$, 更多 I 1 = { 2 } I_1 = \lbrace 2 \rbrace I1​={ 2}

th $2$ iterations:
Solve quadratic programming problem$$\begin{array}{cc}\min & d_1^2+d_2^2-2d_1-4d_2\\ s.t.& -d_1=0\end{array}$$得$d=[0,2{]}^T\ne 0$, 则计算 α = min ⁡ i ∈ { 1 , 3 } { b i − a i T x 0 a i T d ∣ a i T d > 0 } = 1 2 < 1 \alpha = \underset{i \in \lbrace 1, 3 \rbrace}{\min} \left \lbrace \dfrac{b_i - \boldsymbol{a}_i^{\rm T} \boldsymbol{x}_0}{\boldsymbol{a}_i^{\rm T} \boldsymbol{d}} \mid \boldsymbol{a}_i^{\rm T} \boldsymbol{d} > 0 \right \rbrace = \dfrac{1}{2} < 1 a=i∈{ 1,3}min​{ aiT​dbi​−aiT​x0​​∣aiT​d>0}=21​<1, $i=1$, update I 2 = { 1 , 2 } I_2 = \lbrace 1, 2 \rbrace I2​={ 1,2}, $x_2=[0,1{]}^T$

th $3$ iterations:
Solve the quadratic programming problem$$\begin{array}{cc}\min & d_1^2+d_2^2-2d_1-4d_2\\ s.t.& d_1+d_2=0\\ & -d_1=0\end{array}$$得$d=[0,0{]}^{\mathrm{T}}$, $l=[2,0,0{]}^T\ge 0$, 迭代结结

So the optimal solution is$[0,1{]}^T$, 上优值为$1$。

4. Penalty function method and obstacle function method

4.1 Penalty function method

For the general form of constrained optimization problems, the penalty function method converts the problem into an unconstrained optimization problem by adding a penalty term, so that the original problem can be solved using the unconstrained optimization method. For the general form of constrained optimization problem$$\begin{array}{ccc}\min & f(\mathbf{x})& \\ s.t.& h_i(\mathbf{x})=0& i=1,2,\cdots ,l\\ & h_j\left(x\right)\le 0& j=l+1,l+2,\cdots ,m\end{array}$$The solution steps of the penalty function method are:

1. Set$r_1>0$, precision$e>0$, first starting point${\mathbf{x}}_0$, current iteration number$k=1$
2. th$k$ iterations, solve the unconstrained optimization problem min ⁡ P ( x , ρ k ) = f ( x ) + ρ k [ ∑ i = 1 l h i 2 ( x ) + ∑ j = l + 1 m ( max ⁡ { 0 , h j ( x ) } ) 2 ] \min P(\boldsymbol{x}, \rho_k) = f(\boldsymbol{ x}) + \rho_k \left[ \sum_{i=1}^{l}h_i^2(\boldsymbol{x}) + \sum_{j=l+1}^{m} (\max \left \ lbrace 0, h_j(\boldsymbol{x}) \right \rbrace)^2 \right] minP(x,rk​)=f(x)+rk​ ​i=1∑l​hi2​(x)+j=l+1∑m​(max{ 0,hj​(x)})2 ​The optimal solution is $x_k$
3. 若惩罚项满足$$r_k\left[\sum _{i=1}^l{h}_i^2({\mathbf{x}}_k)+\sum _{j=l+1}^m(\max \left\{0,h_j(x_k)\right\}{)}^2\right]\le \epsilon$$ then the iteration ends, and the optimal solution is $x_k$, else$r_{k+1}>r_k$, continue to iterate

Generally speaking, when solving the problem, after iterating once, let$r\to +\infty$Immediately available.

[Example 5] Use penalty function method to solve$$\begin{array}{cc}\min & x_1^2+x_2^2\\ s.t.& x_1-1\ge 0\\ & x_1+x_2=3\end{array}$$【解】令 P ( x 1 , x 2 , ρ ) = x 1 2 + x 2 2 + ρ ( x 1 + x 2 − 3 ) 2 + ρ ( max ⁡ { 0 , 1 − x 1 ] } ) 2 = { x 1 2 + x 2 2 + ρ ( x 1 + x 2 − 3 ) 2 , x 1 > 1 x 1 2 + x 2 2 + ρ ( x 1 + x 2 − 3 ) 2 + ρ ( 1 − x 1 ) 2 , x 1 ≤ 1 \begin{align} P(x_1, x_2, \rho) & = x_1^2 + x_2^2 + \rho \left( x_1 + x_2 - 3 \right)^2 + \rho \left( \max \lbrace 0, 1 - x_1] \rbrace \right)^2 \nonumber \\ & = \begin{cases} x_1^2 + x_2^2 + \rho \left( x_1 + x_2 - 3 \right)^2, & x_1 > 1 \\ x_1^2 + x_2^2 + \rho \left( x_1 + x_2 - 3 \right)^2 + \rho(1 - x_1)^2, & x_1 \le 1 \end{cases} \nonumber \end{align} P(x1​,x2​,ρ)​=x12​+x22​+r(x1​+x2​−3)2+r(max{ 0,1−x1​]})2={ x12​+x22​+r(x1​+x2​−3)2,x12​+x22​+r(x1​+x2​−3)2+ρ(1−x1​)2,​x1​>1x1​≤1​​则$$\frac{\partial P}{\partial x_1}=\{\begin{array}{ll}2x_1+2\rho (x_1+x_2-3),& x_1>1\\ 2x_1+2\rho (x_1+x_2-3)-2\rho (1-x_1),& x_1\le 1\end{array}$$$$\frac{\partial P}{\partial x_2}=2x_2+2\rho (x_1+x_2-3)$$令$$\frac{\partial P}{\partial x_1}=\frac{\partial P}{\partial x_2}=0$$解得$$x=\{\begin{array}{ll}{\left(\frac{3\rho }{2\rho +1},\frac{3\rho }{2\rho +1}\right)}^{\mathrm{T}},& r>1\\ \frac{\rho }{r^2+3\rho +1}{\left(\rho +4,2\rho +3\right)}^T,& 0<r\le 1\end{array}$$So the optimal solution to the original problem is$$\underset{\rho \to +\infty }{\lim }x=\underset{\rho \to +\infty }{\lim }{\left(\frac{3\rho }{2\rho +1},\frac{3\rho }{2\rho +1}\right)}^T={\left(\frac{3}{2},\frac{3}{2}\right)}^T$$Most 优值为$\frac{9}{2}$。

4.2 Obstacle function method

For constrained optimization problems with only inequality constraints$$\begin{array}{ccc}\min & f(\mathbf{x})& \\ s.t.& h_i(\mathbf{x})\le 0& i=1,2,\cdots ,m\end{array}$$The barrier function method constructs a barrier function$b\left(x\right)$General transfer Promise improvement problem, disability function$b\left(x\right)$Usually present and below Construction method:$$b_1\left(x\right)=-\sum _{i=1}^m\frac{1}{h_i\left(x\right)},b_2\left(x\right)=-\sum _{i=1}^m\ln \left[-h_i(\mathbf{x})\right]$$称$b_1\left(x\right)$increase number impairment function, $b_2\left(x\right)$ is the logarithmic barrier function. The solution steps of the obstacle function method are

1. 给定$r_1>0$, precision$e>0$, first starting point${\mathbf{x}}_0$, current iteration number$k=1$
2. th$k$ iterations, solve the unconstrained optimization problem$$\min B(\mathbf{x},r_k)=f\left(x\right)+r_{k}b\left(x\right)$$Get the best solution $x_k$
3. If the penalty term satisfies$$r_{k}b({\mathbf{x}}_k)\le \epsilon$$ then the iteration ends, and the optimal solution is $x_k$, not take$r_{k+1}\in (0,r_k)$, 继续迭代

Generally speaking, when solving the problem, after iterating once, let$r\to 0^{+}$It is the best solution that can be obtained immediately.

[Example 6] Solve by obstacle function method$$\begin{array}{cc}\min & x_1^2+x_2^2\\ s.t.& x_1-x_2+1\le 0\end{array}$$【解】令$$B(x_1,x_2,r)=x_1^2+x_2^2-r\ln (x_2-x_1-1)$$则$$\frac{\partial B}{\partial x_1}=2x_1+\frac{r}{x_2+x_1-1},\frac{\partial B}{\partial x_2}=2x_2-\frac{r}{x_2+x_1-1}$$令$$\frac{\partial B}{\partial x_1}=\frac{\partial B}{\partial x_2}=0$$解得$$x={\left(-\frac{1+\sqrt{1+r}}{4},\frac{1+\sqrt{1+r}}{4}\right)}^T$$So the optimal solution to the original problem is$$\underset{r\to 0^{+}}{\lim }x=\underset{r\to 0}{\lim }{\left(-\frac{1+\sqrt{1+r}}{4},\frac{1+\sqrt{1+r}}{4}\right)}^{\mathrm{T}}={\left(-\frac{1}{2},\frac{1}{2}\right)}^T$$Most 优值为$\frac{1}{2}$。

4.3 Mixed penalty function method

The hybrid penalty function method comprehensively uses the penalty function and the obstacle function, and the objective function is$$F(\mathbf{x},r)=f(x)++rb(\mathbf{x})+\frac{p(\mathbf{x})}{r}$$in expression, $b\left(x\right)$impairment function, < /span>$p\left(x\right)$为惩罚function.

5. Augmented Lagrangian function method

For the general form of constrained optimization problem$$\begin{array}{ccc}\min & f\left(x\right)& \\ s.t.& h_i(\mathbf{x})=0& i=1,2,\cdots ,l\\ & h_j\left(x\right)\le 0& j=l+1,l+2,\cdots ,m\end{array}$$定义其增广拉格朗日函数为 L σ ( x , λ ) = f ( x ) + ∑ i = 1 l λ i h i ( x ) + σ 2 ∑ i = 1 l h i 2 ( x ) + 1 2 σ ∑ j = l + 1 m { [ max ⁡ { 0 , λ j + σ h j ( x ) } ] 2 − λ j 2 } L_\sigma(\boldsymbol{x}, \boldsymbol{\lambda}) = f(\boldsymbol{x}) + \sum_{i=1}^{l}\lambda_i h_i(\boldsymbol{x}) + \dfrac{\sigma}{2}\sum_{i=1}^{l}h_i^2(\boldsymbol{x}) + \dfrac{1}{2\sigma} \sum_{j=l+1}^{m} \left \lbrace \left[ \max \left \lbrace 0, \lambda_j + \sigma h_j(\boldsymbol{x}) \right \rbrace \right]^2 - \lambda_j^2 \right \rbrace Lσ​(x,λ)=f(x)+i=1∑l​li​hi​(x)+2σ​i=1∑l​hi2​(x)+2σ1​j=l+1∑m​{ [max{ 0,lj​+σhj​(x)}]2−lj2​}第$k$ iterations, let$${\nabla }_{\mathbf{x}}L(\mathbf{x},\lambda )=0$$解得${\mathbf{x}}_k$, the infinitesimal infinitesimal equation ( λ k + 1 ) i = { ( λ k ) i + σ h i ( x k ) , 1 ≤ i ≤ l max ⁡ { 0 , ( λ k ) i + σ h i ( x k ) } , l < i ≤ m (\ballsymbol{\lambda}_{k+1})_i = \begin{cases}(\ballsymbol{\lambda}_k)_i + \sigma h_i(\ballsymbol{x}_k), & 1 \le i \le l \\ \max \lbrace 0, (\ballsymbol{\lambda}_k)_i + \sigma h_i(\ballsymbol{x}_k) \rbrace, & l < i \le m \end{cases}(λk+1​)i​={ (λk​)i​+σhi​(xk​),max{ 0,(λk​)i​+σhi​(xk​)},​1≤i≤ll<i≤m​Generally speaking, when solving the problem, iterate once and then calculate $l_k$The optimal solution can be obtained by substituting the limit value into the limit.

[Example 7] Use the augmented Lagrangian function method to solve$$\begin{array}{cc}\min & 3x_1^2+x_2^2\\ s.t.& x_1+x_2=1\end{array}$$【解】令$$L_{\sigma }(x_1,x_2,\lambda )=3x_1^2+x_2^2+\lambda (x_1+x_2-1)+\frac{\sigma }{2}(x_1+x_2-1{)}^2$$则$${\nabla }_x{L}_{\sigma }(x_1,x_2,\lambda )=[6x_1+l+\sigma (x_1+x_2-1),2x_2+l+\sigma (x_1+x_2-1){]}^{\mathrm{T}}$$令$${\nabla }_x{L}_{\sigma }(x_1,x_2,\lambda )=0$$Solution$$x_k={\left[\frac{p-l_k}{4\sigma +6},\frac{3\sigma -3{\lambda }_k}{4\sigma +6}\right]}^T$$Define$$l_{k+1}=l_k+p\left(\frac{p-l_k}{4\sigma +6}+\frac{3\sigma -3{\lambda }_k}{4\sigma +6}-1\right)=\frac{3({\lambda }_k-\sigma )}{2\sigma +3}$$当$l_1>-\frac{3}{2}$When , it is easy to prove by mathematical induction$l_k>-\frac{3}{2}$, given$$l_{k+1}-l_k=\frac{3({\lambda }_k-\sigma )}{2\sigma +3}-l_k=-\frac{\sigma }{2\sigma +3}(3+2{\lambda }_k)<0$$即数列 { λ k } \lbrace \lambda_k \rbrace { λk​}单调递减且圔圔，因为$l_k$The limit of exists, set to $\gamma$, 对$l_k$Taking the limit on both sides of the recursive expression of at the same time, we get$$c=\frac{3(\gamma -\sigma )}{2\sigma +3}$$Interpretation$$c=-\frac{3\sigma }{2\sigma }=-\frac{3}{2}$$Substitute a smooth equation$$\underset{k\to +\infty }{\lim }{\mathbf{x}}_k={\left[\frac{p-\gamma }{4\sigma +6},\frac{3\sigma -3\gamma }{4\sigma +6}\right]}^{\mathrm{T}}={\left[\frac{1}{4},\frac{3}{4}\right]}^T$$Most 优值为$\frac{3}{4}$。