[Optimization]——Introduction to optimization (1)

Preface

This series of articles serves as relevant notes for optimal learning. Bibliography: "Optimization: Modeling, Algorithms and Theory" by Teacher Wen Zaiwen

Optimization problem summary

General form of optimization problem

The optimization problem can generally be described as:
$$\begin{array}{c}\begin{array}{r}\min f(x),\\ \text{s.t.}x\in \mathcal{X},\end{array}\end{array}$$
where $x=(x_1,x_2,\cdot \cdot \cdot ,x_n{)}^T\in {\mathbb{R}}^n$ is the decision variable,$f:R_n\to R$ is the objective function,$\mathcal{X}\subseteq R_n$is a constraint set or feasible region, and the points contained in the feasible region are called feasible solutions or feasible points. The notation st is the abbreviation of "subject to", which refers specifically to constraints. When $\mathcal{X}={\mathbb{R}}_n$When , problem (1) is called an unconstrained optimization problem. Among all decision variables that satisfy the constraint conditions, the variable x ∗ x^∗ that minimizes the objective function$x^{\ast }$ is called the optimal solution of optimization problem (1), that is, for any$x\in X$ has$f(x)⩾f(x^{\ast })$．

PS: Notice that in the set $On\; X$ , function$The\; minimum\; (maximum)\; value\; of\; f$ does not necessarily exist, but its lower (upper) bound"$\inf f(\sup f)$ "always exists. Therefore, when the minimum (maximum) value of the objective function does not exist, we care about its lower (upper) bound, that is, " min ⁡ ( max ⁡ ) \min in question (1$\min \left(\max \right)$ "change to"$\inf (\sup )$"．

Types of optimization problems

• When both the objective function and the constraint function are linear functions, linear programming ;
• When at least one of the objective function and the constraint function is a nonlinear function, the corresponding problem is called nonlinear programming ;
• If the objective function is a quadratic function and the constraint function is a linear function, it is called quadratic programming ;
• Problems containing non-smooth functions are called non-smooth optimization ; smooth functions are functions that are continuously differentiable to infinite orders within its domain, such as exponential functions.
• Problems that cannot be directly derived are called derivative-free optimization ;
• Problems in which variables can only take on integers are called integer programming ;
• The problem of minimizing a linear function about a positive semidefinite matrix under linear constraints is called semidefinite programming , and its generalized form is cone programming .
• According to the properties of the optimal solution:
  • Problems in which the optimal solution has only a small number of non-zero elements are called sparse optimization ;
  • The problem in which the optimal solution is a low-rank matrix is ​​called low-rank matrix optimization.

In addition, there are geometric optimization, quadratic cone programming, tensor optimization, robust optimization, global optimization, combinatorial optimization, network planning, stochastic optimization, dynamic programming, constrained optimization with differential equations, differential manifold constrained optimization, distributed optimization, etc. .

Example: Sparse Optimization

Consider the problem of solving a system of linear equations:

$$\begin{array}{c}\begin{array}{r}Ax=b\end{array}\end{array}$$
where vector $x\in {\mathbb{R}}^n$，$b\in R^m$ , matrix$A\in R^{m\times n}$ , and the vectorbbThe dimension of$b$ is much smaller than the vector xxThe dimension of$x$$m\ll n$ . The system of equations is underdetermined, so there are infinitely many solutions. Ifthe prior information ofsparsity$A$ and the solution to the original problem$u$ satisfies certain conditions, then we can solve$u$ is distinguished from other solutions of the system of equations (2). This type of technology is widely used in compressive sensing, a solution that recovers all information from part of the information.

Features such as sparsity can theoretically guarantee $u$ is the only solution to the system of equations (2) with the fewest non-zero elements, that is,$u$ is as follows$l_0$The optimal solution to the norm problem:

$$\begin{array}{c}\begin{array}{r}\begin{array}{rl}\underset{x\in {\mathbb{R}}^n}{\min }& \Vert x{\Vert }_0,\\ \text{s.t.}& Ax=b.\end{array}\end{array}\end{array}$$
where $\Vert x{\Vert }_0$refers to xxThe number of non-zero elements in$x\; .$Since$\Vert x{\Vert }_0$is a discontinuous function, and its value can only be an integer. Problem (2) is actually NP (non-deterministic polynomial) difficult, and it is very difficult to solve. By converting to $\Vert x{\Vert }_1$problem, you can solve it:
$$\begin{array}{c}\begin{array}{r}\begin{array}{rl}\underset{x\in {\mathbb{R}}^n}{\min }& \Vert x{\Vert }_1,\\ \text{s.t.}& Ax=b.\end{array}\end{array}\end{array}$$
However, it cannot be solved when converted into 2-norm.

Reason:
Geometrically, the three optimization problems actually want to find the smallest $C$ , such that the "norm ball"{ x ∣ ‖ x ‖ ⩽ C } \{x|‖x‖ ⩽ C\}{ x∣‖x‖⩽C}（$\Vert \cdot \Vert$ represents any norm) is exactly the same as$Ax=bIntersect$ ._ The following figure is a schematic diagram:

PS: A norm sphere refers to a spherical set in a certain vector space, with a certain vector as the center and a certain norm as the radius.

for ${\ell }_0$Norm, when $C=2$ { x ∣ ‖ x ‖ 0 ⩽ C } \ {x|‖x‖0 ⩽ C\}{ x∣‖x‖0⩽C } is a full plane, which is naturally related to$Ax=b$ intersects, and when$C=When\; 1$ , it degenerates into two straight lines (coordinate axes). The solution to the problem at this time is$Ax=b$ and the intersection of these two straight lines; for${\ell }_1$norm, according to $C$ is different{ x ∣ ‖ x ‖ 1 ⩽ C } \{x|‖x‖1 ⩽ C\}{ x∣‖x‖1⩽C } is a series of squares, the vertices of these squares happen to be on the coordinate axis, and the smallestCCThe square and straight line A x = b Ax = bcorresponding to$C$$Ax=The\; intersection\; points\; of\; b$ are generally vertices, so${\ell }_1$The solution to the norm is sparse; for ${\ell }_2$norm, when $When\; C$ takes different values​​{ x ∣ ‖ x ‖ 2 ⩽ C } \{x|‖x‖2 ⩽ C\}{ x∣‖x‖2⩽C } is a series of circles, and the circles have smooth boundaries, which are consistent with the straight line$Ax=The\; tangent\; point\; of\; b$ can be any point on the circumference, so${\ell }_2$Norm optimization problems generally cannot guarantee the sparsity of solutions.

$l_0$A norm is not a norm, and this term is used here for the sake of unity of description.
Norm is a concept in mathematics that measures the size or length of a vector. Commonly used norms include Euclidean norm, Manhattan norm, Chebyshev norm, etc.

Let $x$ is$n-$ dimensional vector$(x_1,x_2,\cdots ,x_n{)}^{\top }$ , thenxxNorm of$x$
$\Vert x\Vert$ is defined as:

• Euclidean norm (2-norm) : ${\Vert x\Vert }_2=\sqrt{x_1^2+x_2^2+\cdots +x_n^2}$；
• Manhattan norm (1-norm) : ${\Vert x\Vert }_1=\sum _{i=1}^n|x_i|$；
• Chebyshev norm : ${\Vert x\Vert }_{\infty }=\underset{1\le i\le n}{\max }|x_i|$。

Properties of norms include:

• Non-negativity : $\Vert x\Vert \ge 0$，且 $\Vert x\Vert =0$ if and only if$x$ is a zero vector;
• Homogeneity or homogeneity : $\Vert \alpha x\Vert =|\alpha |\Vert x\Vert$ , where$\alpha$ is a scalar quantity;
• 三角不等式：$\Vert x+y\Vert \le \Vert x\Vert +\Vert y\Vert$；
• 自反性：$\Vert x\Vert =\underset{\Vert y\Vert \le 1}{\max }{x}^{\top }y$(for Chebyshev norm), or
  $\Vert x\Vert =\underset{p\to \infty }{\lim }{\left(\sum _{i=1}^n|x_i{|}^p\right)}^{\frac{1}{p}}$(for other norms).

${\ell }_1$Optimization problem of norm regular term :
$$\begin{array}{c}\begin{array}{r}\underset{x\in {\mathbb{R}}^n}{\min }\mu \Vert x{\Vert }_1+\frac{1}{2}\Vert Ax-b{\Vert }_2^2,\end{array}\end{array}$$

where $m>0$ is the given regularization parameter. Problem (5) is also calledLASSO (least absolute shrinkage and selection operator). This problem can be regarded asthe quadratic penalty functionform of problem (3). Since it is an unconstrained optimization problem, it appears to be simpler than problem (3) in form.

Basic concepts of optimization

Global and local optimal solutions

Before solving the optimization problem, we first introduce the definition of the optimal solution of the minimization problem (1).

If a point is a local minimum solution but not a strict local minimum solution, it is called a non-strict local minimum solution.

Optimization algorithm convergence

Explicit solution : For an optimization problem, if we can use an algebraic expression to give its optimal solution, then this solution is called an explicit solution , and the corresponding problem is often relatively simple.

However, practical problems often cannot be solved explicitly, so iterative algorithms are often used.

• The basic idea of ​​the iterative algorithm is: from an initial point $x_0$Start by iterating according to a given rule and get a sequence $x_k$．If the iteration terminates within a finite number of steps , then the last point is the solution to the optimization problem. If the iteration point sequence is an infinite set , then it is hoped that the limit point (or convergence point ) of the sequence will be the solution to the optimization problem.
• For a specific algorithm, depending on its design point, we are not sure that we can obtain a high-precision approximate solution. At this time, in order to avoid unusable computational overhead, we also need some convergence criteria to stop the algorithm in time.
• In algorithm design, an important criterion is whether the point sequence generated by the algorithm converges to the solution of the optimization problem .
• Consider the unconstrained case. For an algorithm, given the initial point $x_0$, record the points produced by the iteration as {xk} \{x_k\}{ xk​}．if$\left\{x_k\right\}$ In a certain norm∣$||\cdot$In the sense of$||$${\lim }_{k\to \infty }\Vert x^k-x^{\ast }\Vert =0,$ and the point of convergence$x^{\ast }$ is a local (global) minimum solution, then we say that the point sequence converges to the local (global) minimum solution, and the corresponding algorithm is said toconverge to the local (global) minimum solution according to the point sequence.
• Further, if from any initial point $x_0$Starting from the starting point, the algorithm all converges to the local (global) minimum solution according to the point sequence. We call the algorithm globally converges to the local (global) minimum solution according to the point sequence. Correspondingly, if the corresponding function value sequence is recorded as { f ( xk ) } \{f (x^k)\}{ f(xk )}, we can also define the concept that the algorithm (global) converges to the local (global) minimum according to the function value.
• For convex optimization problems, because any local optimal solution is a global optimal solution , the convergence of the algorithm is relative to its global minimum .
• In addition to the convergence of point sequences and function values, commonly used in practice are the optimality conditions of each iteration point (such as the gradient norm in unconstrained optimization problems, the violation of optimality conditions in constrained optimization problems, etc. ) convergence of
• For the constrained case , given the initial point $x_0$, the point sequence { xk } \{x_k\} generated by the algorithm{ xk​} is not necessarily feasible (i.e.{ xk } \{x_k\}{ xk​} ∈ $X$ may not be suitable for any$k$ is established). Considering the constraint violation situation, we need to ensure that$\left\{x_k\right\}$ After converging to$x^{\ast }$ , its violation is acceptable. Apart from this requirement, the definition of convergence of the algorithm is the same as for the unconstrained case.

When designing optimization algorithms, we have some basic guidelines or techniques. For complex optimization problems, the basic idea is to transform them into a series of simple optimization problems (the optimal solutions of which are easy to calculate or have explicit expressions) to solve step by step. Commonly used techniques include:

Asymptotic convergence speed of the algorithm

QQ in dots$Q$ - Convergence speed ($Q$ means "quotient") as an example (function value$Q$ - Convergence rate can be defined similarly). Let{ xk } \{x_k\}{ xk​} is the iterative point sequence generated by the algorithm and converges to$x^{\ast }$.

Except $In\; addition\; to\; Q$ - convergence speed, another commonly used concept is$R$ - Convergence rate ($R$ means "root")

algorithm complexity

A concept closely related to the convergence speed is the complexity of the optimization algorithm $N\left(\epsilon \right)$ , that is, calculate the given accuracyε εThe number of iterations or floating-point operations required for the solution of$\epsilon \; .$