[Convex Optimization Study Notes 1] What is optimization, the mathematical expression of optimization, and the classification of optimization problems

Reference materials:
1. Teacher Ling Qing’s Convex Optimization Course (B station)
2. Stephen Boyd’s "Convex Optimization" Chinese translation (Tsinghua University Press)

Article directory

What is Optimization

Optimization is also called Mathematical Programming.
Optimization: Find the optimal element from a set of feasible solutions.
Three elements are indispensable: the set of feasible solutions, the method of finding, and the optimal elements.

Mathematical Expression Form of Optimization Problem

$\begin{array}{ll} \operatorname{minimize} & f_{0}(x) \\ \text {subject to} & f_{i}(x) \leqslant b_{i}, \quad i=1, \cdots, m \end{array}$

where:
$x$ is calledthe optimization variable(Optimization Variable), which is a vector $x=\left(x_{1}, \cdots, x_{n}\right)$
$f_{0}(x)$ is calledthe objective function(Objective Function), which is a mapping from n-dimensional space to 1-dimensional space, namely $f_{0}: \mathbf{R}^{n} \rightarrow \ mathbf{R}$ 。
$f_{i}(x)$ is a set ofinequality constraints(Inequality Constraint), $f_{i}: \mathbf{R}^{n} \rightarrow \mathbf{R}$

Our goal is to find an $x^{\star}$ , is anoptimal solution(Optimal).
If $x^{\star}$ is an Optimal, then it is equivalent to:
there exists any vector $z$ , satisfy all constraints $z\in\{ f_{1}(z) \leqslant b_{1}, \cdots, f_{m}(z) \leqslant b_{m}\}$ , there are: $f_{0}(z) \geqslant f_{0}\left(x^{\star}\right)$

上面 $\{ f_{1}(z) \leqslant b_{1}, \cdots, f_{m}(z) \leqslant b_{m}\}$ set is calleda feasible solution set(feasible set).
The optimal solution may also have an optimal solution set, that is, there is more than one optimal solution.

Example 1

Take the unary function as an example, the figure:
insert image description here

Find the minimum value of the function in the interval [-1,1].

Converted to an optimization problem:
Objective function: $f_{0}(x)=x^2+1$
Feasible solution set: $x\leqslant1,-x\leqslant1$
Find the optimal solution (the solution is $x = 0$ ）

Example 2 (data fitting problem)

There is a set of sample points (two-dimensional coordinate system as an example), and a line needs to be fitted.
insert image description here

We let this line be a binomial $y=ax^2+bx+c$ , the coefficients a, b, and c of each item are the parameters to be estimated, set the vector $w = (a, b, c)$ 。

目标函数： $f_{0}(w)=\sum_{\mathrm{i}=1}^{\mathrm{n}} [y_i-(ax_i^2+bx_i+c)]^2$ ， $i$ is each sample point.
Finding the optimal solution is a typical least squares problem.

Classification of Optimization Problems

Linear programming, nonlinear programming

If in the optimization problem, the objective function and constraint function $f_0,...,f_m$ All are linear functions, that is, for any $\in\mathbf{R}^{n}$ 和 $\alpha,\beta\in\mathbf{R}$ ， have:
$f_{i}(\alpha x+\beta y)=\alpha f_{i}(x)+\beta f_{i}(y),i=0,1,...,m$
Then such an optimization problem is calleda linear programming.
if $f_0,...,f_m$ If one of them is not a linear function, then the problem is called nonlinear programming .

Give an example of a linear program:

insert image description here
The pentagon in the figure is composed of 5 linear functions $f_1,...,f_5$ The feasible solution set constituted, because the constraint functions are all linear functions, so the feasible solution set must grow into such a pentagonal shape.
At the same time, the objective function $f_0$ It is also a linear function. When we put the objective function in the space, it will be presented with different contour lines to represent its value.
If the arrow direction is $f_0$ The direction of descent, then its optimal solution is on the marked black dot.

Convex programming, non-convex programming

People used to define linear programming and nonlinear programming problems as simple and difficult problems, but later found that it is better to define convex programming as simple problems and non-convex programming as difficult problems.
Any linear programming problem must be convex. Convex programming can be seen as an extension of linear programming.

If both the objective function and the constraint function are convex functions, that is, for any $\in\mathbf{R}^{n}$ 和 $\alpha,\beta\in\mathbf{R}$ ,one value $\alpha+\beta=1,\alpha\geqslant0,\beta\geqslant0$ ，电影不等式官方：
$f_{i}(\alpha x+\beta y)\leqslant\alpha f_{i}(x)+\beta f_{i}(y),i=0,1,...,m$
A function such as $m is a convex function.$
The problem in which both the objective function and the constraint function are convex functions is a convex programming problem, also called a convex optimization problem.

It can be seen that convexity is a more general property than linearity. Because it only needs to satisfy the inequality. Only a value restriction of α and β is added.

Other categories

①Smooth optimization problem, non-smooth optimization problem

smooth and non-smooth for the objective function $f_0$ In terms of.
Smooth function means objective function $f_0$ All points on the domain are differentiable.

② Continuous optimization problem, discontinuous optimization problem

For the feasible domain.
If the feasible region is continuous, it is a continuous optimization problem (such as the pentagon diagram above).
The feasible region is discrete, which means it is a discontinuous optimization problem. Discrete problems are generally hard problems.

③Single-objective problem, multi-objective problem

For multi-objective problems, say we want to optimize two objective functions:
$\operatorname{minimize} f_{1}(x), f_{2}(x)$
This is the multi-objective problem.

For $f_{1}, f_{2}$ , we can often plot two function values:
insert image description here

This line is also known as the Pareto front. It is difficult for us to find a point to make the two optimal. Generally, we need to find a compromise point.
So we generally need to weight the two to convert multi-objective into a single-objective problem:
$\operatorname{minimize} \alpha_1 f_{1}(x) +\alpha_2 f_{2}(x)$
This is only a workaround and may not necessarily work.