Introduction to Convex Optimization CH1

CH1 Introduction

convex optimization

minimize $f_0(x)$

subject to $f_i(x)\leq b_i,i = 1,2,...m$

$f_0(x):R^n->R$, the objective function,$f_i(x):R^n->R$, the constraint function

Linear programming: any$x,y\in R^n,\alpha, \beta \in R$, have$f_i(\alpha x+ \beta y) = \alpha f_i(x) + \beta f_i(y)$

Convex Optimization: Any$x,y\in R^n,\alpha, \beta \in R$, and satisfy$\alpha+\beta = 1, \alpha \geq 0, \beta \geq 0,$, the following inequality holds$f_i(\alpha x+ \beta y) \leq \alpha f_i(x) + \beta f_i(y)$

Least squares, linear programming are special convex optimization problems. There are many efficient algorithms for solving convex optimization problems, and in some cases it can be shown that interior point methods can solve these convex optimization problems with given accuracy in polynomial time.

One point worth mentioning is that by adding a regular term to the least squares cost function, when$x)$Adding a penalty when it is too large can make the resulting solution more realistic than just optimizing the first term. In statistical estimation, when$x$When the distribution of is known in advance, a regularization method can be used. Least squares of quadratic regularization terms:$\sum_{i=1}^k(a_i^Tx-b_i)^2+\rho\sum_{i=1}^nx_i^2$

Conceptually, a problem can be solved quickly if it can be formulated as a convex optimization problem. The key is to judge whether the problem is convex optimization and to transform the problem into convex optimization.

nonlinear optimization

local optimization

Instead of searching for the optimal feasible solution that minimizes the value of the objective function, a local optimal solution (satisfactory solution) is sought. Only the objective function and constraint function are required to be differentiable, and local optimization can be quickly solved. Disadvantages are:
- Unable to estimate how far the local optimum is from the global optimum
- Sensitive to initial values ​​- Sensitive to
parameter values

Local optimization needs to select a suitable algorithm, adjust the parameters of the algorithm, select a good enough initial point, or provide a method for selecting a better initial point. Local optimization methods are an effective technique and not just a technique. Unlike convex optimization, modeling a real problem as a nonlinear optimization problem is fairly straightforward, and the trick is mostly in the solution.

Global optimization

Global optimization can find the absolute worst parameter value if, often used for worst-case analysis problems and verification problems in high-value systems and safety-first systems. If the system performance is still acceptable in the worst case, then the system is safe and reliable.

Application of convex optimization in non-convex optimization (omitted)

Using Convex Optimization to Find Initial Values ​​in Local Optimization

Heuristics in Nonconvex Optimization

Bounds for Global Optimization