实用线性代数和凸优化 Convex Optimization

If not specified, the following conditions are assumed.
$$\begin{gathered} X\in R^{n\ast m} \\ A\in R^{m\ast n} \end{gathered}$$

Trace

$$\begin{gathered} tr(A)=\sum _i{a}_{ii} \\ tr(A+B)=tr(A)+tr(B) \\ tr(cA)=c\cdot tr(A) \\ tr(A)=tr(A^T) \\ tr(A^{T}B)=tr(A{B}^T)=\sum _{i,j}(A\circ B{)}_{ij} \\ tr(b{a}^T)=a^{T}b \end{gathered}$$

Derivation

$$f(x)=||x-p|{|}_2->\nabla f(x)=\frac{x-p}{||x-p|{|}_2}$$

Vector

Subspace

$$x,y\in V;\alpha x+\beta y\in V$$

Basis

a set B of vecs of min cardinality s.t. span(B) = S

Norms

$$\begin{gathered} ||x||\ge 0 \\ ||x+y||\le ||x||+||y|| \\ ||cx||=|c|\cdot ||x|| \end{gathered}$$

Cauchy-Schwarz Inequality

$$|x^{T}y|\le ||x|{|}_p||y|{|}_q;\forall \frac{1}{p}+\frac{1}{q}=1$$

Theorems

Orthogonal Theorem

$$X=S\oplus S^{\perp };\forall S\subset X$$

Projection Theorem

$$mi{n}_{x\in S}||y-x||\Rightarrow y^{\ast }\in S,(y-y^{\ast })\perp S$$

Matrix

Partition

$$\begin{gathered} Ab=\sum _j{a}_j{b}_j \\ {c}^{T}A=\sum _i{c}_i{A}_i \end{gathered}$$

Range

R ( A ) = { A x : x ∈ R n } R m = R ( A ) ⊕ N ( A T ) R n = R ( A T ) ⊕ N ( A ) R(A) = \{Ax: x \in R^n\} \\ R^m = R(A) \oplus N(A^T) \\ R^n = R(A^T) \oplus N(A) \\ R(A)={ Ax:x∈Rn}Rm=R(A)⊕N(AT)Rn=R(AT)⊕N(A)

Fundamental Theorem of Linear Algebra

$$w=Ax+z,z\in N(A^T)$$

Kernel

$$K=X^{T}X$$

Orthogonal

$$A{A}^T=A^{T}A=I,A^T=A^{-1}$$

Schur Complements

$$\begin{gathered} M=\left[\begin{array}{cc}A& X\\ X^T& B\end{array}\right] \\ S=A-X{B}^{-1}{X}^T \\ M\succcurlyeq 0⟺S\ge 0 \end{gathered}$$

Positive Definiteness

For a symmetric square matrix A, PSD means
$$x^{T}Ax\ge 0,\forall x\in R^m⟺{\lambda }_i(A)\ge 0$$
The determinant of PSD is non-negative. The numbers on the diagonal are non-negative.

The definition of PD replaces all $\ge$ to $>$.

If a PSD matrix is invertible, then it is PD.

Matrix Norms

Frobenius Norm

$$||A|{|}_F=\sqrt{tr(A{A}^T)}=\sqrt{\sum _{i,j}|A_{ij}{|}^2}=\sqrt{\sum _i{\lambda }_i(A{A}^T)}$$

Operator Norm

$$||A|{|}_p=ma{x}_{||u|{|}_p=1}||Au|{|}_p$$

l1 norm: largest abs col sum

l2 norm: $\sqrt{{\lambda }_{max}(A{A}^T)}={\sigma }_1$

l_inf norm: largest abs row sum

Nuclear Norm

$$||A|{|}_{\ast }=\sum _i{\sigma }_i$$

Matrix Decomposition

Orthogonal-Triangular Decomposition (QR)

$$A=QR$$

For square matrix A, Q is orthogonal, R is upper triangular.

For non-square matrix with m < n, we still have $Q^{T}Q=I_m$. It can be useful to partition both Q and R.

Cholesky Decomposition

A is PD, L is lower triangular matrix
$$A=L{L}^T$$

Singular Value Decomposition (SVD)

For non-zero matrix A,
$$\begin{gathered} A=U\Sigma V^T \\ A{v}_i={\sigma }_i{u}_i,A^T{u}_i={\sigma }_i{v}_i,i=1\sim r \\ {\sigma }_i^2={\lambda }_i(A{A}^T)={\lambda }_i(A^{T}A),i=1\sim r \\ ||A|{|}_F^2=tr(A^{T}A)=\sum _{i=1}^n{\sigma }_i^2 \\ {A}_k=\tilde{U}\tilde{\Sigma }{\tilde{V}}^T \\ varianceexplained={\eta }_k=\frac{||A_k|{|}_F^2}{||A|{|}_F^2}=\frac{{\sigma }_1^2+...+{\sigma }_k^2}{{\sigma }_1^2+...+{\sigma }_n^2} \end{gathered}$$
$u_i$ and $v_i$ are eigen vectors of $A^{T}A$ and $A{A}^T$ respectively.

$x_j$ in low dimension is ${\tilde{x}}_j=\tilde{S}{\tilde{V}}^T{e}_j$, to recover use $x_j^{\prime }=\tilde{U}{\tilde{x}}_j$.

Spectral Decomposition

A is a square symmetric matrix.
$$A=U\Lambda U^T=\sum _i{\lambda }_i{u}_i{u}_i^T$$
Rayleigh quotient
$${\lambda }_{min}\le \frac{x^{T}Ax}{x^{T}x}\le {\lambda }_{max},x\ne 0$$
For any matrix, Matrix gain (spectral norm)
$$||A|{|}_2=ma{x}_{||x|{|}_2=1}||Ax|{|}_2\le \sqrt{{\lambda }_{max}(A^{T}A)}$$

Sample Covariance Matrix

$$\begin{gathered} C=\frac{1}{m}\sum _{i=1}^m(x_i-\hat{x})(x_i-\hat{x}{)}^T \\ {s}_i=w^T{x}_i \\ {\sigma }^2=\sum _{i=1}^m(w^T{x}_i-\hat{s})=w^{T}Cw \\ tr(C)=\frac{1}{m}||X|{|}_F^2 \end{gathered}$$

Apparently, the covariance matrix is PSD.

Ellipsoid

Let P be a PD matrix, such that $P=L{L}^T=U\Lambda U^T$. The standard form is
E = { x : x T P − 1 x ≤ 1 } E = \{x: x^T P^{-1} x \le 1\} \\ E={ x:xTP−1x≤1}
Converting another form to the standard form.
E = { x ^ + L z : ∣ ∣ z ∣ ∣ 2 ≤ 1 } = { x : ∣ ∣ L − 1 ( x − x ^ ) ∣ ∣ 2 ≤ 1 } = { x : ( x − x ^ ) T P − 1 ( x − x ^ ) ≤ 1 } E = \{\hat x+Lz: ||z||_2 \le 1\} \\ = \{x: ||L^{-1}(x-\hat x)||_2 \le 1\} \\ = \{x: (x-\hat x)^TP^{-1}(x-\hat x) \le 1\} \\ E={ x^+Lz:∣∣z∣∣2​≤1}={ x:∣∣L−1(x−x^)∣∣2​≤1}={ x:(x−x^)TP−1(x−x^)≤1}

Linear Equation

Systems

$$Ax=y$$

We know that # equations = m, # unknows = n

Overdetermined System: m > n, one solution or none

Underdetermined System: m < n, dim(set of solution) = n-m

Square System: m = n

We can solve the system using SVD.
$$\begin{gathered} A=U\Sigma V^T \\ {x}^{\prime }=V^{T}x,y^{\prime }=U^{T}y \end{gathered}$$
Assume that rank(A) = r < m.
$$\Sigma x^{\prime }=y^{\prime }\Rightarrow y_i^{\prime }=\{\begin{array}{ll}{\sigma }_i{x}_i^{\prime },& i=1\sim r\\ 0,& i=r+1\sim m\end{array}$$
If $y\notin R(A)$, the system is not feasible.

If $y\in R(A)$, the system is feasible, $x_i^{\prime }=y_i^{\prime }/{\sigma }_i,i=1\sim r$.

If A is full column rank, then there is unique solution.

Linear Dynamical System

$$x_{t+1}=A_t{x}_t$$

The system is time invariant if $A_t=A$. It can be extended to include inputs and offset, or to an auto-regressive model.
$$x_{t+1}=A{x}_t+b$$
The steady-state solution is when $t\to \infty$, $(I-A)x_t=b$.

Least Square

Plain

$$\begin{gathered} mi{n}_x||Ax-y|{|}_2^2 \\ {x}^{\ast }=(A^{T}A{)}^{-1}{A}^{T}y \end{gathered}$$

There is another variation which contains weights, but it can be converted.
$$mi{n}_x||W(Ax-y)|{|}_2^2=mi{n}_x||A_{w}x-y_w|{|}_2^2$$

Constrained

$$mi{n}_x||Ax-y|{|}_2^2:Cx=d$$

Define $x^{\prime }$ s.t. $C{x}^{\prime }=d$. The solution set is $x=x^{\prime }+Bz,B\in N(C)$. We can convert the problem to
$$mi{n}_x||A^{\prime }z-y^{\prime }|{|}_2^2:A^{\prime }=AB,y^{\prime }=y-A{x}^{\prime }$$

Penalties

Take L2 norm as an example.
$$\begin{gathered} mi{n}_x||Ax-y|{|}_2^2+\varphi (x) \\ =mi{n}_x||Ax-y|{|}_2^2+\lambda ||x|{|}_2^2 \end{gathered}$$
We can construct a new A and y.
$$\begin{gathered} A^{\prime }=\left[\begin{array}{c}A\\ \sqrt{\lambda }{I}_n\end{array}\right] \\ {y}^{\prime }=\left[\begin{array}{c}y\\ 0_n\end{array}\right] \end{gathered}$$
This way we can get the solution as below.
$$x^{\ast }=(A^{\prime T}{A}^{\prime })A^{\prime T}{y}^{\prime }=(A^{T}A+\lambda I{)}^{-1}{A}^{T}y$$

Convex Optimization

Equality constraints are allowed if they are affine.

Linear Programming (LP)

$$min{c}^{T}x+d:Ax\le b$$

Quadratic Programming (QP)

$$min\frac{1}{2}{x}^{T}Hx+c^{T}x+d:Ax\le b$$

H is a PSD matrix.

If H is PD, then
$$\begin{gathered} f(x)=\frac{1}{2}(x-x^{\ast }{)}^{T}H(x-x^{\ast })+d-\frac{1}{2}{x}^{\ast T}H{x}^{\ast } \\ {x}^{\ast }=-H^{-1}c \end{gathered}$$
If H is PSD, and $c\in R(H)$, then
$$H{x}^{\ast }+c=0$$
Otherwise, the problem is unbounded.

Quadratic Constrained Quadratic Programming (QCQP)

$$min\frac{1}{2}{x}^T{Q}_{0}x+a_0^{T}x:x^T{Q}_{i}x+a_i^{T}x\le b_i$$

Second-Order Cone Programming (SOCP)

$$min{c}^{T}x:||A_{i}x+b_i|{|}_2\le c_i^{T}x+d_i$$

Robust Programming

$$mi{n}_{x}ma{x}_{u\in U}{f}_0(x,u):f_i(x,u)\le 0,\forall u\in U$$

Consider a single inequality with uncertain coefficient vector.
$$a^{T}x\le b,a\in U$$

Scenario Uncertainty

U is finite.
$$ma{x}_{a\in U}{a}^{T}x\le b$$

Box Uncertainty

U = { a : ∣ ∣ a − a ^ ∣ ∣ ∞ ≤ ρ } = { a : a ^ + ρ u : ∣ ∣ u ∣ ∣ ∞ ≤ 1 } m a x a ∈ U a T x = a ^ T x + ρ ∣ ∣ x ∣ ∣ 1 U = \{a: ||a-\hat a||_{\infty} \le \rho \} = \{a: \hat a + \rho u: ||u||_{\infty} \le 1 \} \\ max_{a \in U} a^Tx = \hat a^T x + \rho||x||_1 U={ a:∣∣a−a^∣∣∞​≤ρ}={ a:a^+ρu:∣∣u∣∣∞​≤1}maxa∈U​aTx=a^Tx+ρ∣∣x∣∣1​

Spherical Uncertainty

U = { a : ∣ ∣ a − a ^ ∣ ∣ 2 ≤ ρ } m a x a ∈ U a T x = a ^ T x + ρ ∣ ∣ x ∣ ∣ 2 U = \{a: ||a-\hat a||_{2} \le \rho \} \\ max_{a \in U} a^Tx = \hat a^T x + \rho||x||_2 U={ a:∣∣a−a^∣∣2​≤ρ}maxa∈U​aTx=a^Tx+ρ∣∣x∣∣2​

Ellipsoidal Uncertainty

U = { a : ( a − a ^ ) T P − 1 ( a − a ^ ) ≤ 1 } m a x a ∈ U a T x = a ^ T x + ∣ ∣ R T x ∣ ∣ 2 , P = R T R U = \{a: (a-\hat a)^T P^{-1} (a-\hat a) \le 1 \} \\ max_{a \in U} a^Tx = \hat a^T x + ||R^Tx||_2, P = R^TR U={ a:(a−a^)TP−1(a−a^)≤1}maxa∈U​aTx=a^Tx+∣∣RTx∣∣2​,P=RTR

Convexity

A subset is said to be convex if it contains the line segment between any two points in it.
$$x_1,x_2\in C,\lambda \in [0,1]\Rightarrow \lambda x_1+(1-\lambda )x_2\in C$$
A function f is convex if its domain and range are convex. Convex functions must be $+\infty$ outside their domains.

The epigraph of a function is
e p i f = { ( x , t ) , x ∈ d o m f , t ∈ R : f ( x ) ≤ t } epi f = \{(x,t), x \in dom f, t \in R: f(x) \le t \} epif={ (x,t),x∈domf,t∈R:f(x)≤t}
f is a convex function iff epi f is a convex set.

Optimality

Consider a problem
$$mi{n}_x{f}_o(x):Ax=b$$
We have
$$\nabla f_0(x^{\ast }{)}^T(x-x^{\ast })\ge 0,\forall x\in \mathcal{X}$$
The proof comes from
$$f_0(x)\ge f_0(x^{\ast })+\nabla f_0(x^{\ast }{)}^T(x-x^{\ast })$$
In a convex unconstrained problem with differentiable objective, x is optimal iff $\nabla f_0(x)=0$. If there is an equality constraint of $Ax=b$, then x is optimal iff $Ax=b,\exists v:\nabla f_0(x)+A^{T}v=0$.

Hulls

Given a set of points $P$ in $R^n$.

Linear Hull

$$x=\sum _{i=1}^m{\lambda }_i{x}_i$$

Affine Hull

$$x=\sum _{i=1}^m{\lambda }_i{x}_i:\sum _{i=1}^m{\lambda }_i=1$$

aff P is the smallest affine set containing $P$.

Conic Hull

$$x=\sum _{i=1}^m{\lambda }_i{x}_i:{\lambda }_i\ge 0$$

Convex Hull

$$x=\sum _{i=1}^m{\lambda }_i{x}_i:\sum _{i=1}^m{\lambda }_i=1,{\lambda }_i\ge 0$$

Preserving Convexity

Intersection Rule

The intersection of convex sets is also a convex set, and it holds for infinite families of convex sets. The intersection of halfspaces is also called a polyhedron.

Affine Transformation

An affine mapping of a convex set is still convex.

Pointwise Maximization

If $(f_a{)}_{a\in A}$ is a family of convex functions, and A is a set (not necessarily a convex set), then the pointwise max function is convex.
$$f(x)=ma{x}_{a\in A}{f}_a$$

Partial Minimization

If g(x, y) is convex in x, y and C is convex, then $f(x)=mi{n}_{y\in C}g(x,y)$ is convex. This result trivially extends to partial minimization over a subset of the function’s arguments.

Composition Function

$$x\to f(g(x))$$

If f is convex and increasing, and g is convex, then the function is convex w.r.t to x.

Constraints

Activeness

If $x^{\ast }$ satisfies $f_i(x^{\ast })<0$, then the i-th inequality constraint is inactive (slack) at the optimal solution $x^{\ast }$.

Problem Transformations

An optimization problem can be transformed into an equivalent one

• monotone transformation (scaling, logarithm, squaring)
• change of variables
• addition of slack variables
• epigraphic reformulation
• replacement of equality constraints with inequality ones
• elimination of inactive constraints (safe feature elimination)
• discovering hidden convexity

Duality

Problem Formulation

Consider an optimization problem in standard form
$$\begin{gathered} p^{\ast }=mi{n}_x{f}_0(x) \\ s.t.:f_i(x)\le 0,i=1\sim m \\ {h}_i(x)=0,i=1\sim q \end{gathered}$$
Note that the objective function and the constraints are not necessarily convex.

Lagrangian

Vectors $\lambda$ and $v$ are referred to as Lagrange multipliers.
$$\mathcal{L}(x,\lambda ,v)=f_o(x)+\sum _{i=1}^m{\lambda }_i{f}_i(x)+\sum _{i=1}^q{v}_i{h}_i(x)$$
The primal can be expressed as
$$p^{\ast }=mi{n}_{x}ma{x}_{\lambda \ge 0,v}\mathcal{L}(x,\lambda ,v)$$

Recovering Primal Solutions

If $\mathcal{L}(x,{\lambda }^{\ast },v^{\ast })$ has an unique minimizer, then it is either primal-optimal solution or there is no such solution if it is not primal-feasible.

Weak Duality

The Minimax Inequality is
$$p^{\ast }=mi{n}_{x}ma{x}_{y}F(x,y)\ge ma{x}_{y}mi{n}_{x}F(x,y)=d^{\ast }$$
Therefore, the weak duality indicates that
$$\begin{gathered} g(\lambda ,v)=mi{n}_x\mathcal{L}(x,\lambda ,v) \\ {d}^{\ast }=ma{x}_{\lambda \ge 0,v}g(\lambda ,v)\le p^{\ast } \end{gathered}$$

Strong Duality

The strong duality is achieved when $p^{\ast }=d^{\ast }$.

Sion’s Minimax Theorem

Let X be convex and Y be a compact set (bounded and closed). If F(x,y) is convex over X and concave over Y, then
$$mi{n}_{x}ma{x}_{y}F(x,y)=ma{x}_{y}mi{n}_{x}F(x,y)$$

Slater’s Condition

If the problem is strictly feasible, then strong duality holds. Namely, there exist $x_0\in relintD$ such that $f_i(x_0)<0$.

Karush-Kuhn-Tucker Condition (KKT)

Strong duality holds iff the KKT conditions are satisfied.

Primal feasibility: $f_i(x)\le 0$

Dual feasibility: $\lambda \ge 0$

Complementary slackness: ${\lambda }_i{f}_i(x)=0$

Lagrangian stationarity: ${\nabla }_x\mathcal{L}(x,\lambda )={\nabla }_x{f}_0(x)+{\sum }_{i=1}^m{\lambda }_i{\nabla }_x{f}_i(x)=0$

Reference

• Optimization Models in Engineering (EECS 227 A), Laurent El Ghaoui, University of California Berkeley, Fall 2021