[Optimization] KKT conditions

first-order condition

This section discusses the first-order condition (KKT) in detail,

Sequential Feasible Direction

Definition Let $x^{\prime }$ is a feasible point,{ x ( k ) } \{x^{(k)}\}{ x( k ) }is a feasible sequence, satisfying$x^{(k)}\to x^{\prime }$ , and$\forall k,x^{\left(k\right)}\ne x^{\prime }$ , then$x^{\left(k\right)}-x^{\text{'}}$ for$x^{\prime }$ , expressed as
$$x^{\left(k\right)}-x^{\prime }=d_k{p}^{\left(k\right)}$$
where $\delta_k > 0 $ and$d_k\to 0$ , then$p^{\left(k\right)}$ is a vector of fixed length, called$p^{}$Any gathering pointpp${}^{of\; (}$${}^k$${}^{)}$$p$ is the feasible region$\Omega$ at$x^{\prime }$ , the feasible directions of the sequence at all are denoted as${\mathcal{F}}^{\prime }$。

By definition, a sequence feasible direction determines a feasible sequence.

For example, the feasible point $x^{\prime }=0$ , forΩ 1 = { x ∈ R 2 : x 2 ≥ x 1 2 } \Omega_1 = \{x\in \mathbb{R}^2:x_2\geq x_1^2\}Oh1​={ x∈R2:x2​≥x12​} , then its sequence feasible direction needs to satisfy$p_2>0$ is fine.

And for Ω 2 = { x ∈ R 2 : x 2 = x 1 2 } \Omega_2 = \{x\in \mathbb{R}^2:x_2 = x_1^2\}Oh2​={ x∈R2:x2​=x12​} , the feasible direction of the sequence needs to satisfy
$$p_1\ne 0,p_2=0$$

first-order necessary condition

May wish to set $f$ at the feasible point$x^{}$The set of descending directions at${}^{\prime }$
is D ′ = { p ∈ R n ∣ p T g ′ < 0 } D' = \{p\in \mathbb{R}^n|p^Tg' < 0\}D′={ p∈Rn∣pTg′<0}

Lemma Let $x^{\ast }$ is a local minimum point of the constrained problem, then at$There\; is\; no\; sequence\; feasible\; direction\; at\; x$ ^* is the descending direction, that is,
$${\mathcal{F}}^{\ast }\cap D^{\ast }=\varnothing$$

In other words, the above lemma states that if $x^{\ast }$ is a local minimum point, then the objective function is at$x^{\ast }$ has a non-negative derivative along any feasible direction of the sequence.

Linearizable Feasible Direction

However, the set $F^{\ast }$ is usually not easy to calculate, consider another feasible direction set which is easy to calculate. Constraint function$c_i$at $x^{\prime }$ The first-order Taylor approximation at
$$c_i(x^{\prime }+s)\approx c_i(x^{\prime })+\nabla c_i(x^{\prime }{)}^{T}s$$
defines the point$x^{}$The feasible direction of linearizationat${}^{\prime }$ is to satisfy
$$\begin{array}{r}{p}^T{a}_i^{\prime }=0,i\in \mathcal{E}\\ p^T{a}_i^{\prime }\le 0,i\in \mathcal{I}\end{array}$$
The nonzero vector $p$ , the set of all linearized feasible directions is$F^{\prime }$。

Obviously, if $F^{\text{'}}$ and$F^{’}$ Same, it will be convenient.

Lemma $F^{\prime }\subseteq F^{\prime }$

Prove that if $p\in F^{\prime }$ , then there exists a feasible sequence{ x ( k ) } \{x^{(k)}\}{ x( k ) }satisfy
$$x^{(k)}-x^{\prime }=d_k{p}^{\left(k\right)}$$
where$d_k\to 0,p^{\left(k\right)}\to p$ . Expand the constraint$c_i$at $x^{The}$ Taylor function
$$c_i(x^{(k)})=c_i(x^{\prime })+d_k{p}^{(k)}{a}_i^{\prime }+o\left(d_k\right)$$
optoi$i\in \mathcal{E}$，有 $c_i(x^{(k)})=c_i(x^{\prime })=0$ ;op to$i\in \mathcal{I}$，有 $c_i\left(x^{\left(k\right)}\right)\le c_i\left(x^{\prime }\right)=0$ For example, give
$$\frac{c_i(x^{(k)})}{d_k}=\frac{c_i(x^{\prime })}{d_k}+p^{(k)}{a}_i^{\prime }+\frac{o(d_k)}{d_k}$$
For $k\to \infty$，$p\in F^{\prime }$ , the lemma is proved.

Unfortunately, $F‘\subseteq F^{\text{'}}$ Not necessarily true.

Example Define the set
Ω = { x ∈ R 2 : x 2 ≤ x 1 3 , x 2 ≥ 0 } \Omega = \{x\in\mathbb{R}^2:x_2 \leq x_1^3,x_2\geq 0 \}Oh={ x∈R2:x2​≤x13​,x2​≥0 }
Consider the feasible point$x^{\prime }=(0,0{)}^T$ , linearized feasible direction$p=(-1,0{)}^T\in F^{\prime }$ , obviously there is no feasible direction to converge to$p$ , ie$p\notin F^{\prime }$。

constraint specification

Constraint specification ( constraint quality, CQ ) refers to the guarantee that $F‘=F^{’}$ assumptions. It should be noted that it israre that the constraint specification fails.

Lemma At feasible point $x^{’}$ , if the condition

(1) LCQ：$c_i(x),i\in {\mathcal{A}}^{’}$ is a linear function, or

(2) LICQ：$a_i^{\prime },i\in A^{\text{'}}$ linearly independent

established, then there is $F‘={\mathcal{F}}^{\prime }$。

Farkas Lemma

Farkas Lemma Given $n-$ dimensional vector$a_1,a_2,\cdots ,a_m$and $g$，集合
S = { p ∈ R n : p T g < 0 , p T a i ≤ 0 , i = 1 , 2 , ⋯   , m } S = \{ p \in \mathbb{R}^n: p^Tg<0,p^Ta_i \leq 0, i = 1,2,\cdots,m \} S={ p∈Rn:pTg<0,pT ai​≤0,i=1,2,⋯,m }
is an empty setif and only ifthere exists$l_i\ge 0,i=1,2,\cdots ,m$，使得
$$-g=\sum _{i=1}^m{a}_i{l}_i$$

Given ${\mathbb{R}}^{}$Vectora 1 , a 2 , ⋯ , am a_1,a_2,\cdots,a_m in${}^n$$a_1,a_2,\cdots ,a_m$，令
$$C=\left\{v\in R^n:v=\sum _{i=1}^m{a}_i{l}_i,l_i\ge 0\right\}$$
then$C$ is apolygonal coneand is aclosed convex set,

If the vector $a\in C$ , then there exists a hyperplane separation$Cwaaa$ _$a$ , that is, there exists a non-zero vector$p$ 使得
$$p^{T}a>0,p^{T}v\le 0,\forall v\in C$$

To connect the necessary conditions of the lemma with the Lagrange multipliers, it is necessary to extend the Farkas lemma to the case of equality.

Corollary Given ${\mathbb{R}}^n$ medium$g^{\ast },a_i^{\ast },i\in \mathcal{E},a_i^{\ast },i\in {\mathcal{I}}^{\ast }$，则集合
S = { p ∈ R n : p T g ∗ < 0 , p T a i ∗ = 0 , i ∈ E , p T a i ∗ ≤ 0 , i ∈ I ∗ } S = \{ p \in \mathbb{R}^n: p^Tg^*<0, p^Ta_i^*= 0,i\in \mathcal{E},p^Ta_i^* \leq 0, i \in \mathcal{I}^*\} S={ p∈Rn:pTg∗<0,pT ai∗​=0,i∈E,pT ai∗​≤0,i∈I∗ }
is empty and exists$l_i^{\ast },i\in E,l_i^{\ast }\ge 0,i\in I^{\ast }$，使得
$$-g^{\ast }=\sum _{i\in \mathcal{E}}{l}_i^{\ast }{a}_i^{\ast }+\sum _{i\in {\mathcal{I}}^{\ast }}{l}_i^{\ast }{a}_i^{\ast }$$

The KKT condition can be proved by the above deduction.

Regularity assumption 1 : $F^{\ast }\cap D^{\ast }=F^{\ast }\in D^{\ast }$

If $x^{\ast }$ is a local minimum point, and at$x^{The\; regularity\; assumption\; 1\; at\; \ast }$ holds, then
$$F^{\ast }\cap D^{\ast }=\varnothing According$$
to Farkas lemma, we know that$\exists {\lambda }_i^{\ast }\in {\mathcal{A}}^{\ast }$ , and$l_i^{\ast }\ge 0,i\in {\mathcal{I}}^{\ast }$，使得
$$g^{\ast }+\sum _{i\in \mathcal{E}}{l}_i^{\ast }{a}_i^{\ast }+\sum _{i\in {\mathcal{I}}^{\ast }}{l}_i^{\ast }{a}_i^{\ast }=0$$
当 $i\in \mathcal{I}\backslash {\mathcal{I}}^{\ast }$ 时，有 $c_i(x^{\ast })<0$ , then$l_i^{\ast }=0$。

KKT conditions

Theorem (first-order necessary condition) If $x^{\ast }$ is a local minimum point and$x^{\ast }$ regularity assumption
$$F^{\ast }\cap D^{\ast }=F^{\ast }\cap D^{\ast }$$
holds, then there is a Lagrange multiplier$l^{\ast }$ Make$x^{\ast },l^{\ast }$ 满足
$$\begin{array}{rl}{\nabla }_x\mathcal{L}(x^{\ast },l^{\ast })& =0\\ c_i(x^{\ast })& =0,i\in \mathcal{E}\\ c_i\left(x^{\ast }\right)& \le 0,i\in \mathcal{I}\\ l_i^{\ast }& \ge 0,i\in I\\ l_i^{\ast }{c}_i(x^{\ast })& =0,i\in \mathcal{I}\end{array}$$

The regularity assumption is that for the vector $a^{\prime }(i\in {\mathcal{A}}^{’})$for further relaxation.

Theorem Let $x^{\ast }$ is a local minimum point of the constraint problem, and at$x^{\ast }$ where LCQ or LICQ is established, then$x^{\ast }$ Satisfies KKT conditions..

Example Consider the problem
$$\begin{array}{rl}\min & x_2\\ \mathrm{s}.\mathrm{t}.& x_2\le x_1^3,x_2\ge 0\end{array}\text{\hspace{1em}(1)}$$
与
$$\begin{array}{rl}\min & x_1\\ s.t.& x_2\le x_1^3,x_2\ge 0\end{array}\text{\hspace{1em}( 2 )}$$
In the solution$x^{\ast }=(0,0{)}^{At\; T}$ , it is easy to verify that (1) the regularity assumption is satisfied, and (2) the regularity assumption is not satisfied.

references

[1] Liu Hongying, Xia Yong, Zhou Yongsheng. Fundamentals of Mathematical Programming, Beijing, 2012.