[Optimization basis] Second-order conditions

second order condition

equality constraints

First consider the case of only equality constraints
$$\begin{array}{rl}\underset{x\in {\mathbb{R}}^n}{\min }& f(x)\\ \mathrm{s}.\mathrm{t}.& c_i(x)=0,i=1,2,\cdots ,m\end{array}$$
Suppose the optimal solution $x^{\ast }$ exists, and at point$x^{\ast }$ Directional quantity$a_i^{\ast }(i\in E)$ Linear independence (that is, the LICQ constraint specification holds). For the sequence feasible direction$p\in {\mathcal{F}}^{\ast }$ , there exists a feasible sequence$x^{\left(k\right)}$ and the corresponding direction sequence$p^{(k)}\to$For$p$
$$f(x^{(k)})=f(x^{\ast }+d_k{p}^{\left(k\right)})=\mathcal{L}(x^{\ast }+d_k{p}^{(k)},l^{\ast })$$
because$x^{\ast }$ isL \mathcal{L}The stable point of$L$$\nabla \mathcal{L}=0$ ), so$L$ at$x^{\ast }$ Specialized Taylor equation
$$\begin{array}{rl}f(x^{\ast }+d_k{p}^{(k)})& =\mathcal{L}(x^{\ast }+d_k{p}^{(k)},l^{\ast })\\ & =f^{\ast }+\frac{1}{2}{d}_k^2{p^{(k)}}^T{W}^{\ast }{p}^{(k)}+o(d_k)\end{array}$$
For
$$W^{\ast }={\nabla }_x^2\mathcal{L}(x^{\ast },l^{\ast })={\nabla }^{2}f(x^{\ast })+\sum _{i=1}^m{l}_i^{\ast }{\nabla }^2{c}_i(x^{\ast })$$
means the Lagrange function aboutxxHessian matrix of$x\; .$

For example,if
$$\frac{f(x^{\ast }+d_k{p}^{(k)})}{d_k^2}=\frac{f^{\ast }}{d_k^2}+\frac{1}{2}{p^{(k)}}^T{W}^{\ast }{p}^{\left(k\right)}+\frac{o(d_k)}{d_k^2}$$
And $k\to \infty$ , and$x^{\ast }$ is a local minimum point,
$${p^{\left(k\right)}}^T{W}^{\ast }{p}^{\left(k\right)}\ge 0$$

Theorem (second-order necessary condition) Let $x^{\ast }$ is the local minimum point of the problem and satisfies the KKT condition, and the corresponding Lagrange multiplier is$l^{\ast }$ , then reverse ppis feasible for any sequence$p$ ,if
$$p^T{W}^{\ast }p\ge 0$$
inferenceLet$x^{\ast }$ is the local minimum point of the problem and satisfies the KKT condition, and the corresponding Lagrange multiplier is$l^{\ast }$，若 ${\mathcal{F}}^{\ast }=F^{\ast }$ ,if
$$p^T{W}^{\ast }p\ge 0,\forall p\in F^{\ast }$$

Theorem (second-order sufficient condition) Let $x^{\ast }$ is the KKT point of the problem, and the corresponding Lagrange multiplier is$l^{\ast }$ , if condition
$$p^T{W}^{\ast }p>0,\forall p\in F^{\ast }$$
is established, then$x^{\ast }$ is a strict local minimum of the problem.

Example Consider the problem
$$\begin{array}{rl}\min & f(x)=\frac{1}{2}(x_1-1{)}^2+\frac{1}{2}{x}_2^2\\ \mathrm{s}.\mathrm{t}.& c(x)=x_1-\beta x_2^2=0\end{array}$$
Discussion parameter $When\; \beta$ takes any value,$x^{\ast }=(0,0{)}^T$ is a local minimum point?

Solution Since $g^{\ast }=(-1,0{)}^T$，$a^{\ast }=(1,0{)}^T$ , so the first-order condition is satisfied,$x^{\ast }$ is a KKT point, and$l^{\ast }=1$，$W^{\ast }={\left(\begin{array}{cc}1& 0\\ 0& 1-2\end{array}\right)}^T$

又 F ∗ = { p = ( 0 , p 2 ) T : p 2 ≠ 0 } F^* = \{p=(0,p_2)^T:p_2\neq0\} F∗={ p=(0,p2​)T:p2​​=0 } , so$p^T{W}^{\ast }p=(1-2\beta )p_2^2$. Thus, there are

• 当 $b<\frac{1}{2}$，$x^{\ast }$ is a strict local minimum point
• 当$b>\frac{1}{2}$，$x^{\ast }$ Not a local minimum point
• 当$b=\frac{1}{2}$，$x^{\ast }$ is a strict local minimum point

Weak Positive Constraint vs. Strong Positive Constraint

Let $x^{\ast }$ is a KKT point,$l^{\ast }$ is the corresponding Lagrange multiplier.

Define
KaTeX parse error: Undefined control sequence: \or at position 41: …in\mathcal{E} ~\̲o̲r̲~ \lambda_i^* >…
as a set of strong positive constraints. That is, from ${\mathcal{A}}^{\ast }$ removes weak positive constraints, that is,$l_i^{\ast }=0,i\in {\mathcal{I}}^{\ast }$ , get$A_{+}^{\ast }$。

定义
G ∗ = { p ∈ F ∗ : c i ( x ( k ) ) = 0 , ∀ i ∈ A + ∗ } \mathcal{G}^* = \{p\in\mathcal{F}^*:c_i(x^{(k)}) = 0,\forall i \in \mathcal{A}^*_+\} G∗={ p∈F∗:ci​(x(k))=0,∀i∈A+∗​}

G ∗ = { p ∈ R n : p ≠ 0 , a i ∗ T p = 0 , i ∈ A + ∗ , a i ∗ T p ≤ 0 , i ∈ A ∗ \ A + ∗ } G^* = \{p\in \mathbb{R}^n: p \neq 0, {a_i^*}^Tp = 0, i \in \mathcal{A}^*_+, {a_i^*}^Tp \leq 0, i \in \mathcal{A}^* \backslash \mathcal{A}_+^*\} G∗={ p∈Rn:p​=0,ai∗​Tp=0,i∈A+∗​,ai∗​Tp≤0,i∈A∗\A+∗​}

In fact, there are

• $G^{\ast }\subset G^{\ast }$
• 设 $p\in F^{\ast }$ , then$p^T{g}^{\ast }=0$ if and only if$p\in G^{\ast }$

Regularity assumption 2: $G^{\ast }=G^{\ast }$

general questions

First, the general constraint optimization problem
$$\begin{array}{rl}\underset{x\in {\mathbb{R}}^n}{\min }& f(x)\\ \mathrm{s}.\mathrm{t}.& c_i(x)=0,i\in \mathcal{E}\\ & c_i\left(x\right)\le 0,i\in \mathcal{I}\end{array}$$
There are the following second-order conditions

Theorem (second-order necessary condition) Let $x^{\ast }$ is the local minimum point of the problem and satisfies the KKT condition, and the corresponding Lagrange multiplier is$l^{\ast }$ , if regularization condition 2 holds, there must be
$$p^T{W}^{\ast }p\ge 0,\forall p\in G^{\ast }$$

Theorem (second-order sufficient condition) Let $x^{There\; is\; a\; Lagrange\; multiplier\; at\; \ast }$ so that the KKT condition is true, and the corresponding Lagrange multiplier is$l^{\ast }$ , if condition
$$p^T{W}^{\ast }p>0,\forall p\in G^{\ast }$$
is established, then$x^{\ast }$ is a strict local minimum of the constrained problem.

references

[1] Liu Hongying, Xia Yong, Zhou Yongsheng. Fundamentals of Mathematical Programming, Beijing, 2012.