[Optimization] Related conditions for optimization

Optimal related conditions

---

1. Necessary conditions for local extreme points

Since we discussed local (global) extreme points in the previous section, we also want to know, if a point is an extreme point, what conditions need to be met?

1. Necessary conditions for local minimum point

If point x ₀ is the local minimum point of the function f(x), then:

• x ₀ is the stagnation point of f(x)-(the first derivative of f(x) at x ₀ is 0)
• The Hessian matrix H(x ₀ ) of f(x) at x ₀ is positive semi-definite

2. Proof

[Self Summary]

• Relevant proofs about extreme values ​​often use the definition of extreme values ​​and Taylor expansion
• Disproval method is also a frequently used proof idea
• In order to construct the definition and Taylor expansion required above, the neighborhood of a certain point is often used for corresponding calculations.

3. Necessary conditions for local maxima

Although the previous discussion is based on local minima as an example, we can easily transfer conclusions by analogy to local maxima.

According to the above figure, we can see that if a point x ₀ is a local maximum point of f(x), then there are:

• The value of the first derivative of f(x) at x ₀ is 0
• The Hessian matrix of f(x) at x ₀ is negative semi-definite

---

2. Sufficient conditions for local extreme points

1. Sufficient conditions for local minima

If a point x ₀ on the function f(x) satisfies:

• x ₀ is the stagnation point of f(x)-(the first derivative of f(x) at x ₀ is 0)
• The Hessian matrix H(x ₀ ) of f(x) at x ₀ is positive definite

Then it can be obtained that x ₀ is the strict local minimum point of f(x) (<)

2. Proof

3. Matters needing attention

①Comparing the sufficient conditions and the necessary conditions, it can be seen that the sufficient conditions are more stringent than the necessary conditions (the necessary conditions cover the sufficient conditions)
②The scope of application of the sufficient conditions is narrower, the function needs to be defined on the open set , and the function needs to be second-order differentiable

In practical problems, functions are often defined on closed sets, and the best points are likely to appear on the endpoints.

At this time, we can first use sufficient conditions to find the optimal point of the open set corresponding to the closed set, and then compare it with the function value on the end point to determine the optimal point of the entire closed set.

③Sufficient conditions can be used to directly find all non-singular points in the extreme points, while singular points are more difficult to find

Pay attention to distinguish the relationship between "sufficient conditions and "necessary conditions"

• Sufficient to meet the conditions of points must be non-singular extreme point , the point does not meet the full criteria are not necessarily extreme points
• All extreme points must meet the necessary conditions , and meeting the necessary conditions is not necessarily an extreme point
• Those extreme points that meet the necessary conditions but do not meet the sufficient conditions are called singular points

④ For a differentiable convex function f(x) defined on an open set X , ▽f(x ^* ) = 0↔ point x ^* is the global minimum point of f(x) .

[Brief proof]
According to the theorem 1 of "Three Theorems of Convexity of Differentiable Functions " in the previous blog post "Basic Concepts of Optimization Theory ":

We can take any x ∈ X, then f( x)-f(x ^* )≥▽f(x ^* ) ^T ·(xx ^* );
also because according to the meaning of the question, ▽f(x ^* ) ^T = 0, so f(x)-f(x ^* )≥0

to prove that x ^* is the global minimum point of f(x).

---

[Example] Solve the local extreme points

of a given function. Attention should be paid to the problem of solving extreme points:

• After solving the stagnation point, it must be verified with the second derivative to determine whether it is a minimum (large) value point
• It can only be concluded that the point satisfies the necessary conditions, but it cannot be concluded that the point is a local extreme point.
• Different functions have the same properties at the first and second derivatives of a certain point. This does not mean that the point is a point with the same properties in the two functions. It can be analyzed by calculating higher-order derivatives.

---

3. Conditions for the existence of local extreme points

Not all functions have extreme points:

1. Conditions for the existence of extreme points

In general, we only need to remember to solve in a given bounded interval, generally there are extreme points

2. Discuss the reasons for various conditions of extreme points

In this section, we discussed the sufficiency, necessity, sufficiency, necessity and existence conditions of extreme points.

• Help understand the extremum solution process
• When the conditions are met, we can terminate an iterative algorithm in due course
• Can help optimize the algorithm