Optimization: Modeling, Algorithms and Theory (Optimization Modeling - 1)

Optimization: Modeling, Algorithms and Theory

I am currently studying the book Optimization: Modeling, Algorithms and Theory. I will record it here and take some notes. I will also add some of my own understanding to it and try not to write in such a rigid way (of course the most basic still have)

Chapter 3 Optimization Modeling

This chapter will start with common modeling techniques, and then introduce common optimization models in statistics, signal processing, image processing, and machine learning. We will focus on explaining the ideas and practical implications behind optimization modeling.

3.1 Modeling techniques

3.1.1 Design of objective function

1. Least square method

In my opinion, very few beginners should study this book directly about the least squares method, so everyone should be familiar with this thing.
Let ${\varphi }_i(x):R^n\to R,i=1,2,\cdots ,m$ to$n$ -ary function, and there are the following equations
$$b_i={\varphi }_i(x),i=1,2,\cdots ,m\text{\hspace{1em}( 3.1.1 )}$$
inside$b_i$is a known real number. We know that this problem is not always solvable. First of all, if the number of equations $m$ exceeds the number of independent variables$n$ , so the solution to the system of equations may not exist. Secondly, due to factors such as measurement errors, the equation relationship of the system of equations may not be established accurately. In order to solve it in actual situations, the idea of ​​the least squares method is to minimize the error$l_2$The number square, immediately
$$\underset{x\in R^n}{\min }\sum _{i=1}^m(b_i-{\varphi }_i(x){)}^2\text{\hspace{1em}( 3.1.2 )}$$
If${\varphi }_i\left(x\right)$ is a linear function, then it is called linear least squares, otherwise it is called nonlinear least squares. The
idea of ​​least squares is very intuitive. If there is a solution to equation (3.1.1), then solve problem (3.1. The global optimal solution of 2) is 0, which is equivalent to finding the solution to the equation. If the solution to the equation does not exist, problem (3.1.2) gives a solution with the smallest error to a certain extent.

The least squares method uses $l_2$Norm is used to measure the size of the error. Its main advantages are two
(1) $l_2$The square norm is smooth and differentiable, which will bring better properties to the objective function
(2) $l_2$The norm is optimal for handling certain errors, and the answer will be given later.

Of course, least squares is not always the most reasonable. No free lunch theorem. According to actual problems, we often build least squares models and max-min models. The idea is to use different norms instead of l 2 $l_2$Norm, if you want to ensure the minimum regression of the sum of deviations and absolute values, the corresponding model is:
$$\underset{x\in R^n}{\min }\sum _{i=1}^m|b_i-{\varphi }_i(x)|\text{\hspace{1em}( 3.1.3 )}$$
If you want to minimize the maximum deviation, the corresponding optimization model is:
$$\underset{x\in R^n}{\min }\underset{i}{\max }|b_i-{\varphi }_i(x)|\text{\hspace{1em}(3.1.4)}$$

2. Regularization

When modeling, we often need to rely on the properties of the solution we want to obtain. For example, when there is more than one optimal solution, not all solutions are necessarily what we want. In order to make the solution smooth and overcome the problem The pathological properties of , or in order to solve the problem of over-fitting (more smoothness is actually a disguised solution to over-fitting), then the improved model is
$$\underset{x\in R^n}{\min }\sum _{i=1}^m(b_i-{\varphi }_i(x){)}^2+\mu ||x|{|}_2^2\text{\hspace{1em}( 3.1.5 )}$$
where$m>0$ is a balance parameter. If you want to get a sparse solution, you can use$l_0$The norm constructs the following model ( $l_0$The norm is defined as the number of non-zero elements in the vector)
$$\underset{x\in R^n}{\min }\sum _{i=1}^m(b_i-{\varphi }_i(x){)}^2+\mu ||x|{|}_0\text{\hspace{1em}( 3.1.6 )}$$
where$m>0$ is used to control the sparsity of the solution, but since$l_0$The norm is difficult to handle in practice, so we often use $l_1$norm to ensure sparsity, the model is as follows
$$\underset{x\in R^n}{\min }\sum _{i=1}^m(b_i-{\varphi }_i(x){)}^2+\mu ||x|{|}_1\text{\hspace{1em}(3.1.7)}$$

In image processing, $x$ itself may not be sparse, but it is sparse in the transform domain, and the corresponding model is
$$\underset{x\in R^n}{\min }\sum _{i=1}^m(b_i-{\varphi }_i(x){)}^2+\mu ||W(x)|{|}_0\text{\hspace{1em}(3.1.8)}$$
以及
$$\underset{x\in R^n}{\min }\sum _{i=1}^m(b_i-{\varphi }_i(x){)}^2+\mu ||W(x)|{|}_1\text{\hspace{1em}( 3.1.9 )}$$
where$W:R^n\to R^p$ represents some kind of transformation, commonly used ones include total variation and wavelet transform

The significance of the regular term in the objective function is obvious. We must satisfy the error as small as possible and its coefficient should not be too large. If the coefficient is too large, the penalty of its regular term will be very high, and when all the coefficients If all are reduced, the model will naturally become smooth.

3. Maximum likelihood estimation

In practical problems, many data come from unknown distributions. It is very bug-biting to infer the specific form of the distribution from the data. Maximum likelihood estimation is a method commonly used in statistics to estimate the probability distribution, and through the maximum likelihood function, Make the observed data obey the assumed model as much as possible.

Here we consider a simple situation, assuming that we already know that the data comes from a specific distribution, but we do not know the specific parameters of the distribution, for convenience, let p ( a ; x ) p(a;x $p(a;x)$ is its distribution law or probability density function, where$x$ is an unknown parameter, in order to estimate$x$ , we select a sequence of independent and identically distributed sample points$a_1,a_2,\cdots ,a_n$, the likelihood function is defined as the parameter $x$ , number set { ai , i = 1 , 2 , ⋯ , n a_i,i=1,2,\cdots,n $a_i,i=1,2,\cdots ,n$ } probability of occurrence, that is,
$$L(x)=\prod _{i=1}^{n}p(a_i;x)$$
$L\left(x\right)$ is actually this$The\; joint\; probability\; (joint\; density)\; of\; n$ points, but at this time the independent change becomes the parameter$x$
Then it is now obvious that the maximum likelihood estimate of the parameters is defined as
$$\hat{x}\in \underset{x\in \chi }{\mathrm{argmax}}L\left(x\right)$$
is actually asking for$The\; x\; that\; is\; the\; largest\; of\; L\left(x\right)$ is determined by the parameter$x$ maximizes the probability of this event happening,
where$\chi$ is the parameter space. Assuming that the maximum likelihood estimate exists, solving the maximum likelihood estimate is essentially to find the parameters that are most likely to produce the sample in a family of distributions (similar to what I said above). In practice, the likelihood function The maximum value of the logarithm is easier to solve, that is, consider the maximization problem
$$\underset{x\in \chi }{\max }l(x)=\ln L\left(x\right)\text{\hspace{1em}( 3.1.10 )}$$
Because$ln\left(x\right)$ is strictly monotonically increasing. In actual calculations, I think the biggest reason may be that multiplication can be converted into addition, which is easier to operate.

4. Cost, loss, and gain functions

Many problems in operations research are the process of minimizing costs (losses) and maximizing benefits. For example, when we play a game, there will be a corresponding score reward for every step we take. We naturally expect the higher the final score. The better. When traveling, we hope to have the shortest route or the least travel expenses when visiting all cities. When pricing items in supermarkets, we usually determine the most beneficial pricing based on the price and its corresponding possible sales number. In these In practical problems, they can all be written in the form of optimization problems. The objective function is either to minimize losses, maximize benefits, or both (minimize risk, maximize benefits).

5. Functional, variation

Many problems in physics and chemistry can be expressed in the form of energy minimization. For example, in the calculation of electronic structure, we calculate the stable state by minimizing the interaction energy between atoms and electrons. Generally speaking, the energy is generally The function is defined on the function space, that is, the independent variable of the corresponding optimization problem is a function in the infinite-dimensional space. We can obtain its corresponding optimality conditions through variation. Another method commonly used in practice is Using appropriate discretization, the minimization problem of the energy functional is pulled back from the infinite dimensional space to the finite space, thereby obtaining the discrete solution of the corresponding problem. Note that different discretizations generally correspond to different objective functions and different optimization problem.
PS: I haven’t learned this yet, so I can’t quite understand it in depth.

6. Slack problem

When the original problem is not easy to solve, a commonly used technique in optimization is relaxation. The basic idea of ​​this technique is to use simple terms to replace the difficult-to-handle terms in the objective function while retaining some of the properties of the original problem, so that Problems are easier to solve. For example, $l_0$The norm is non-differentiable and non-convex, since $l_1$The norm is $l_0$The norm is approximate to some extent. In practice, $l_1$Norm instead of $l_0$norm, since $l_1$The norm is convex, and the theoretical analysis and algorithm design of its corresponding model will be simpler.
For low-rank optimization problems, the rank corresponds to the number of non-zero elements in the singular values ​​of the matrix, which is also non-convex and non-differentiable. The common way is to use the kernel norm of the matrix (l 1 l_1 of the vector composed of the singular values ​​of the $l_1$norm) instead, resulting in a more tractable convex relaxation term

Another slack strategy is e.g. dealing with a $minf\left(x\right)$ problem, using a lower bound of the objective function$f_R\left(x\right)$ to replacef ( x ) f(x)Solve for$f$$($$x$$)$$f_R\left(x\right)$ should satisfy:
(1)$f_R(x)\le f(x),\forall x\in \chi$
（2）$f_R\left(x\right)$ has a simple structure
. The Lagrangian function introduced later can actually be regarded as the relaxation of the objective function of the original problem. It should be noted
that the problem after relaxation is not necessarily equivalent to the original problem. We have repeatedly mentioned$l_0$The norm can be replaced by $l_1$Norms are performed under certain conditions, $l_2$The norm generally cannot be taken as $l_0$norm relaxation

3.1.2 Design of constraints

1. The physical nature of the problem itself

Depending on the actual significance of the problem, the decision variables of the optimization problem need to satisfy various constraints. For example, in electronic structure design, we assume that orbital functions are orthogonal to each other; also in the design of aircraft wings, Affected by the airflow around the wing, the change in wing shape corresponds to a differential equation and in many cases non-negative constraints. We also relax isochronous constraints into inequality constraints when linear or general equality observations are noisy or more robustness is needed.

2. Equivalent substitution

For an optimization problem, if the objective function is in the form of a composite function, such as:
$$\underset{x}{\min }f(Ax+b)$$
We often introduce the variable$y$ and the equality constraint$y=Ax+bConsidering$ the equivalent problem with constraints, it becomes:
$$\underset{x}{\min }f(y)s.t.y=Ax+bFor$$
the optimization problem${\min }_{x}f(x)=h(x)+r\left(x\right)$ , we often introduce$y$ and the constraint$x=y$ , convert it into
$$\underset{x}{\min }h(x)+r(y)s.t.x=In\; this\; way,\; y$$
splits the objective function. For inequality constraints, we can also convert them into equality constraints and simple non-negative or non-positive constraints by introducing slack variables. For example, consider the constraint c ( x ) ≤$c\left(x\right)\le 0$ , pull-in$y\ge 0$ , which can be equivalently converted into
$$c(x)+y=0,y\ge 0$$
In addition, we can also use the above figure to get the equivalent form of the problem, for the optimization problem${\min }_{x}f\left(x\right)$ , according to the definition of the figure above, it can be seen that it is equivalent to the optimization problem
$$\underset{x,t}{\min }t,s.t.f\left(x\right)\le t$$
Equivalent conversion of the constraints of the optimization problem will help us study the mathematical properties of the problem more conveniently. After equivalent conversion, many problems may become the same type of problem, and then we can Find a unified algorithm or model to solve.

3. Relaxation

When the constraints of the original optimization model are too complex, we can also use relaxation techniques to replace the intractable constraints with easily tractable constraints. The feasible region of the relaxed problem will be larger than the original problem. For example, the box constraint $x\in [0,1]$ instead of the integer constraintx ∈ { 0 , 1 } x{\in}\{0,1\}x∈{ 0,1 } , or use the inequality constraint$c\left(x\right)\ge 0$ instead of the equality constraint$c(x)=0$
There are two basic principles to follow when enlarging the feasible region:
(1) Simplify the original constraints, that is, the problem after relaxation must be easier to handle, otherwise why relax it?
(2) Don’t make the feasible region too large. , over-enlarging the feasible region will lose the key information of the original problem, and it will be meaningless to solve it.

There is a very natural question, what is the connection between the solution of the relaxation problem and the solution of the original problem? Generally speaking, the relaxed problem is not equivalent to the original problem, but under certain conditions it can be proved that the solution of the relaxed problem is the solution of the original problem.

3.2 Regression analysis

3.2.1 Overview

The general regression model can be written in the following form
$$b=f(a)+ϵ\text{\hspace{1em}(3.2.1)}$$
其中$a\in R^d$ is the independent variable,$b\in R$ is the response variable,$ϵ\in R$ is the error (noise) of the model. In practical problems, we generally only know$a$ and$The\; observed\; value\; of\; b$ , and the error is unknown, the regression model is established using$m$个观测值($a_i,b_i$) to find $The\; specific\; form\; of\; f$ , and then make predictions on the response variable through the newly observed independent variable.

How to choose $f$ is very important, of course, we can of course make a model let$f(a_i)=b_i$All are correct, but if $If\; f$ is very complex, the generalization ability will be relatively poor, which is overfitting. Of course, it can’t be too different. A good model needs to take into account two aspects. There is a small error in the observed data, and at the same time it has a simple form.

3.2.2 Linear regression model

设$(x_i,y_i),i=1,2,\cdots ,m$ is the observed independent variable and response variable, and different data points are independent of each other, then for each data point (the independent variables and response variables in the book are always written in the form a, b, w, etc., which makes me uncomfortable, I wrote it according to my previous habit) The coefficient of the non-constant term here is$n-1$ is for convenience, after adding the constant term, the total is$n$个好看一点
$$y=w_1{x}_i+w_2{x}_{i2}+\cdots +w_{n-1}{x}_{in-1}+b+ϵ,i=1,2,\cdots ,m$$
inside$w_i$is the parameter that needs to be determined. The input feature plus the constant term $x_i=(x_{i}1{)}^T$，令$w=(w_1,w_2,\cdots ,w_{n-1},b{)}^T\in R^n$ , then the linear regression model can be written as
$$y_i=w^T{x}_i+{ϵ}_i\text{\hspace{1em}( 3.2.2 )}$$
If we want to write it in matrix form, we set
$$X=\left[\begin{array}{c}{x}_1^T\\ x_2^T\\ x_3^T\\ ⋮\\ x_m^T\end{array}\right]$$
$$y=\left[\begin{array}{c}{y}_1\\ y_2\\ y_3\\ ⋮\\ y_m\end{array}\right]$$
$$ϵ=\left[\begin{array}{c}{ϵ}_1\\ {ϵ}_2\\ {ϵ}_3\\ ⋮\\ {ϵ}_m\end{array}\right]$$
(I don’t know how to combine them at the moment, sorry)
$X$ is$m\ast$matrix of$n$$yandϵ$ are$m\ast 1$，$w$ is$n\ast 1$ , we get the matrix form
$$y=Xw+ϵ\text{\hspace{1em}( 3.2.3 )}$$
Now we have to consider how to solve this linear regression model. We don’t want to take it for granted here to minimize the error. We use maximum likelihood to do it, which is more mathematical.

Assume ${ϵ}_i$是高斯白噪声，即${ϵ}_i\sim N（0，{\sigma }^2）$，那么根据${ϵ}_i=y_i-w^T{x}_i$，我们有
$$p({ϵ}_i)=p(y_i|x_i;w)=\frac{1}{\sqrt{2\pi {\sigma }^2}}exp(-\frac{(y_i-w^T{x}_i{)}^2}{2{\sigma }^2})$$
其对数似然函数为
$$l(x)=ln\prod _{i=1}^{m}p({and}_i|x_i;w)=-\frac{m}{2}ln(2\pi )-mln\sigma -\sum _{i=1}^m\frac{(y_i-w^T{x}_i{)}^2}{2p^2}$$
The maximum likelihood estimation is to maximize the likelihood function. After removing the constant term, we get the following least squares problem
$$\underset{x\in R^n}{\min }\frac{1}{2}||Wx-y|{|}_2^2\text{\hspace{1em}( 3.2.4 )}$$ Note that ϵ i \epsilon_i
does not need to be known when constructing the maximum likelihood estimate${ϵ}_i$The most important thing
above is to establish the connection between solving the regression model and the least squares method. When the error is assumed to be Gaussian white noise, the least squares solution you solve is the solution of the linear regression model. Of course, if ϵ i
\ ${ϵ}_i$If it is not subject to Gaussian white noise, the solution to the linear regression model is not the solution to the least squares model. It may be the solution to the least one multiplication problem under certain noises, etc. I don’t know if everyone can understand clearly

3.2.3 Regularized linear regression model

Regularization is widely used in regression models. For example, when the number of features in a data set is greater than the number of samples, the solution to problem 3.2.4 is not unique, and it is necessary to use regularization items to select solutions with different properties.

1. Tikhonov regularization

To balance the fitting properties of the model with the smoothness of the solution, Tikhonov regularization or ridge regression adds $l_2$The square of the norm is a regular term, assuming ${ϵ}_i$is Gaussian white noise, then it has $l_2$The linear regression of the norm square regular term actually solves the following problem:
$$\underset{w\in R^n}{\min }\frac{1}{2}||Wx-y|{|}_2^2+\mu ||w|{|}_2^2\text{\hspace{1em}( 3.2.6 )}$$
Due to the existence of regular terms, the objective function of this problem is a strongly convex function, which is actually for$w$ for punishment, another common deformation is to give the parameter$p>0$，求解
$$\underset{w\in R^n}{\min }\frac{1}{2}||Wx-y|{|}_2^{2}s.t.||w|{|}_2\le \sigma \text{\hspace{1em}( 3.2.7 )}$$
当$\mu$和$When\; \sigma$ satisfies a certain relationship, their solutions can be the same

2.Lasso problem and deformation

If you want the solution to be sparse, you can consider adding $l_{!}$The norm is a regular term, and the LASSO regression problem is as follows
$$\underset{w\in R^n}{\min }\frac{1}{2}||Wx-y|{|}_2^2+\mu ||w|{|}_1$$
The specific reason is emm. You can draw a picture to understand it. The LASSO model plays the function of feature extraction.
Similar to question 3.2.7, you can also consider the problem
$$\underset{w\in R^n}{\min }\frac{1}{2}||Wx-y|{|}_2^{2}s.t.||w|{|}_1\le \sigma \text{\hspace{1em}( 3.2.8 )}$$
Considering the noise$The\; existence\; of\; ϵ$ can also be given$v>0$，考虑模型
$$\underset{w\in R^n}{\min }||w|{|}_{1}s.t.||Wx-y||\le v\text{\hspace{1em}( 3.2.9 )}$$
The essential ideas of optimization models 3.2.8 and 3.2.9 are similar, that is, "under the condition of controlling the error,$w$ is$l_1$"Norm should be as small as possible", but the types of optimization problems they belong to are actually different. We will further explain it later in Chapter 4.

Of course, if $If\; ϵ$ is not Gaussian white noise, the loss function needs to be selected according to the specific type.
The traditional LASSO problem requires$w$ is a sparse solution, but the sparsity of the actual problem may be expressed in many ways. If asked$w$ has a grouped sparse solution, i.e.$The\; components\; of\; x$ can be divided into G groups, and the parameters in each group must be 0 or non-0 at the same time. The solution of the traditional LASSO problem cannot meet such requirements. Therefore, a grouped LASSO model is proposed: min ⁡
$$\underset{x\in R^n}{\min }\frac{1}{2}||Ax-b|{|}_2^2+m\sum _{i=1}^G\sqrt{n_l}||w_{I_l}|{|}_2\text{\hspace{1em}( 3.1.12 )}$$
Part$I_l$Belongs to the llthIndicator set of$l$
$$|I_l|=n_l，\sum _{l=1}^G{n}_l=n$$
当$n_l=1,l=1,2,\cdots ,When\; G$ , 3.1.12 degenerates into the traditional LASSO problem. It can be seen that the regular term is$||x_{I_l}||$ l$l_1$范数，分组LASSO问题把稀疏性从单个特征提升到了组的级别上，但不要求组内的稀疏性。
如果既要保证分组的稀疏性，也要保证单个特征的稀疏性，可以考虑将两种正则项结合起来，like this:
$$\underset{x\in R^n}{\min }\frac{1}{2}||Ax-b|{|}_2^2+{\mu }_1\sum _{i=1}^G\sqrt{n_l}||w_{I_l}|{|}_2+{\mu }_2||w|{|}_1\text{\hspace{1em}(3.1.13)}$$

对于某些实际问题，特征$w$ itself is not sparse, but it is sparse under a certain transformation, so we also need to adjust the corresponding regularization term. The general problem form can be:
$$\underset{w\in R^n}{\min }\frac{1}{2}||Wx-y|{|}_2^2+\mu ||Fw|{|}_1\text{\hspace{1em}( 3.2.14 )}$$
Of course, this transformation is sparse and can also be used with$w$ itself is sparsely combined, so that a sparse solution can also be obtained, andwwThe changes between the components of$w\; are\; relatively\; gentle.$

3.3 Logistic regression

In a classification problem, the output variable takes a value in a discrete space. For a two-class classification problem, there are only two predictor variables, namely -1 and 1. In the past, the most common ones are actually label 0 and 1. I believe that both It’s commonplace, so let’s introduce -1 and 1 here to see if it will be easier. We introduce one of the most classic and basic classification models: the logistic regression model, given the feature $x$ , logistic regression assumes that the probability that this sample belongs to category 1
$$p(1|x;w)=P(and=1|x;w)=\theta \left(w^{T}x\right)$$
sigmoid$sigmoid$函数
$$\theta \left(z\right)=\frac{1}{1+exp(-z)}$$
Then the probability of belonging to category -1
$$p(-1|x;w)=1-p(1|x;w)=\theta (-w^{T}x)$$
Therefore, for the above problem, we can simply write it as
$$p(y|x;w)=\theta (y\ast w^{T}x)$$
Assume the data pair$x_i,y_i,i=1,2,\cdots ,m$ are independently and identically distributed, then given$x_1,x_2,\cdots ,x_m$情况下，$y_1,y_2,\cdots ,y_m$的联合概率密度是
$$p({and}_1,y_2,\cdots ,y_m|x_1,x_2,\cdots ,x_m;w)=\prod _{i=1}^{m}p({and}_i|x_i;w)=\frac{1}{{\prod }_{i=1}^m(1+exp(-y_i\ast w^{T}x))}$$
In fact, it is to find the maximum probability, and then take $ln$ , changed together with the negative sign, the maximum likelihood estimation is to solve the following model:
$$\underset{w\in R^n}{\min }\sum _{i=1}^{m}ln(1+exp(-y_i\ast w^{T}x))\text{\hspace{1em}( 3.3.2 )}$$
Of course, you can also add a series of regular terms after this model, such as
$$\underset{w\in R^n}{\min }\sum _{i=1}^{m}ln(1+exp(-y_i\ast w^{T}x))+\lambda ||w|{|}_2^2\text{\hspace{1em}(3.3.3)}$$

The result number is $x_i,y_i$is a pair of random variables $\{a,\beta \}$ , then the loss function can be written in the mean form
$$E\left[ln\right(1+exp(-b\ast a^{T}w))]$$
and the regular term can be written as the following abstract form:
$$\underset{w\in R^n}{\min }E\left[ln\right(1+exp(-b\ast a^{T}w))]+\lambda r(x)$$
where$r\left(x\right)$ is the regular term,$\lambda$ is a regular parameter. In fact, many machine learning models can be written as more general stochastic optimization problems and its discrete version
$$\underset{w\in R^n}{\min }f(x)+\lambda r(x)\text{\hspace{1em}(3.3.5)}$$

where
$$f\left(x\right)=E[F(x,\xi )]$$
Specifically, it is the integral form or the direct mean, depending on whether the loss function is continuous or discrete.

3.4 Support vector machine

Support Vector Machine (SVM) is another widely used binary classification model. Let us first consider a simple case, assuming that the training data set is linearly separable. As shown in the figure, it is easy to see that given a linearly

separable After setting the data, the hyperplane that meets the partitioning requirements is generally not unique. So what is the ideal hyperplane? It should have the following characteristics: the data points are relatively far away from this plane, so it will have good robustness. a little xx
in space$x$ to hyperplane$w^{T}x+b=$The distance d from$0$
$$d=\frac{|w^{T}x+b|}{||w|{|}_2}$$
For a sample point (xi, yi), y ∈ { − 1, 1 } (x_i,y_i),y{\in}\{-1,1\}(xi​,yi​),y∈{ −1,1 } , if he classifies correctly, then
$$y_i(w^T{x}_i+b)>0,i=1,2,\cdots ,m$$
In order to find the ideal hyperplane, we hope that the points in the two types of data go to the hyperplane$w^{T}x+b=$The minimum distanceof$0$ should be as large as possible, and the following initial model can be established
$$\underset{w,b,c}{\max }cs.t.\frac{y_i(w^T{x}_i+b)}{||w|{|}_2}\ge \gamma ,i=\mathrm{1.2.}\cdots ,m\text{\hspace{1em}(3.4.1)}$$
$\gamma$ is the minimum distance from all sample points to the hyperplane, and the goal is to maximize it.
Note that the constraints in 3.4.1 are equivalent to
$$y_i(w^T{x}_i+b)\ge \gamma ||w|{|}_2$$

Now you can think about it, $Wandb$ are scaled by the same positive multiple, which will certainly not affect our overall solution? Just like$w=1,b=2$ legs$w=2,b=The\; straight\; lines\; represented\; by\; 4$ are the same. Therefore, if we do not fix$||w|{|}_2$If you solve it, you will get many solutions that represent the same straight line. This is not necessary, as long as we can ensure that the line we find is that one. Therefore, for convenience, we force $||w|{|}_2=\frac{1}{c}$, then this problem is equivalent to
$$\underset{x,y}{\min }\frac{1}{2}||w|{|}_2^{2}s.t.y_i(w^T{x}_i+b)\ge 1,i=1,2,\cdots ,m\text{\hspace{1em}(3.4.2)}$$

My name can be used $y_i\ast (w^T{x}_i+b)=1$ established$x_i$is called a support vector. It is not difficult to find that the parameters w, bw, b of the hyperplane$wandb$ are completely determined by support vectors.

When the assumption of linear separability does not hold, we introduce a non-negative slack variable $\xi$ , allowing misclassified points, then the constraints in 3.4.1 become
$$\frac{y_i(w^T{x}_i+b)}{||w|{|}_2}\ge \gamma (1-X_i),\xi \ge 0,i=1,2,\cdots ,m$$

Here use $c(1-X_i)$ to represent the distance of the misclassified point. Obviously, the misclassified point should not be too large. We use the function of the slack variable${\sum }_{i=1}^m{X}_i$to control, finally we can get
$$\underset{w,b,x}{\min }\frac{1}{2}||x|{|}_2^2+m\sum _{i=1}^m{X}_{i}s.t.y_i(w^T{x}_i+b)\ge 1-x,X_i\ge 0,i=1,2,\cdots ,m\text{\hspace{1em}( 3.4.3 )}$$
where$\mu$ is the penalty coefficient, increasing$\mu$ will increase the penalty for misclassification

Model 3.4.3 is also equivalent to the unconstrained optimization problem
min ⁡ w , b 1 2 ∣ ∣ x ∣ ∣ 2 2 + μ ∑ i = 1 mmax { 1 − yi ( w T xi + b ) , 0 } (3.4. 4) \min_{w,b}\frac{1}{2}||x||_2^2+{\mu}{\sum_{i=1}^m}max\{1-y_i(w^ Tx_i+b),0\}\tag{3.4.4}w,bmin​21​∣∣x∣∣22​+mi=1∑m​max{ 1−yi​(wTxi​+b),0}(3.4.4)

We see that $max{1-y_i(w^Tx_i+b),0}$ is the pair that does not satisfy the inequality $y_i(w^T{x}_i+b)$ , if you are not satisfied with the penalty, then it is a positive number. If it is, it is 0. This can also be regarded as the penalty function method of 3.4.2, although max { z , 0 } max\{ z,0\}max{ z,0 } is not differentiable, but its simple form also provides many possibilities for algorithm design. It can also be seen that introducing different penalty functions can construct different SVM models.
In addition, when there are redundant features in the training data, people also consider$||x|{|}_2^2$Replace with $l1$ norm

3.5 Probabilistic graphical model

The probability graphical model is an important concept in probability theory. It is a probability model that uses a graph structure to describe the conditional independent relationship between multivariate random variables.
Given a random vector $X=(X_1,X_2,X_3,\cdots ,X_n)$ , the corresponding joint probability is$n$ -ary function, according to the conditional probability,
$$P(X=x)=\prod _{k=1}^{n}P(X_k=x_k|X_1=x_{!},\cdots ,X_{k-1}=x_{k-1})$$
Assume that each variable$X_i，i=1,2,\cdots ,n$ is discrete, and each has$m$ values, then without any independence assumption, we need$(m^n-1)$ parameters can be used to determine its probability distribution. I thought about it for a long time when I saw this. I don’t know why it is this number. So, let’s give an example first (you can skip it if you understand it) assuming X 1
,$X_1,X_2,X_3$The three variables are binary (0,1) distributions. When the interdependence relationship is not known, if the probability distribution is calculated,
$$P(X=x)=P(X_1=x_1)P(X_2=x_2|X_1=x_1\left)P\right(X_3=x_3|X_1=x_1,X_2=x_2)$$
大家这时候想一下，如果我们要知道X的概率分布，我们要知道多少个参数（概率）呢？
要知道$P(X_1=0)$和$P(X_1=1)$
$P(X_2=0|X_1=0)，P(X_2=1|X_1=0),P(X_2=0|X_1=0),P(X_2=1|X_1=1)$
$P(X_3=0|X_1=0,X_2=0)，P(X_3=1|X_1=0,X_2=0),..........$
这里我们乍一看很多对把，但是其实你只要知道$P(X_1=0)$那自然就知道$P(X_1=1)$了吧？所以对于$P(X_1=x_1)$我们只需要一个参数
那按照这个思想，对于$P(X_2=x_2|X_1=x_1)$总共需要两个参数，$P(X_3=x_3|X_1=x_1,X_2=x_2)$需要四个参数，总共需要1+2+4=7个参数
好那现在我们来推一下，对于$n$元函数，每个$X_i$有$m$个取值，为什么需要$m^n-1$个参数呢？
依照刚刚的想法，$P(X_1=x_1)$需要$m-1$个参数，$P(X_2=x_2|X_1=x_1)$需要$(m-1)\ast m$个参数
$P(X_n|X_1=x_1,\cdots ,X_{n-1}=x_{n-1})$需要$(m-1)m^{n-1}$ parameters
Then a total of$m-1+(m-1)m+\cdots +(m-1)m^{n-1}$ , mention$(m-1)$ , and then wait for the summation (this will always be the case), and then you can get$m^n-1$ la

Saying so much is actually to calculate the assumption that when certain variables are known, there will be independence between variables. Going back to the first example, assume that $X_2$When, $X_1$and $X_3$独立，则有
$$P(X=x)=P(X_1=x_1)P(X_2=x_2|X_1=x_1\left)P\right(X_3=x_3|X_1=x_1,X_2=x_2)$$
$$=P(X_1=x_1)P(X_2=x_2|X_1=x_1\left)P\right(X_3=x_3|X_1=x_1)$$
According to what we said before, only five parameters (two less) are needed to determine the joint distribution

When there are many variables in the probability model, their corresponding dependencies will also be more complex. At this time, the graphical model can help us understand the conditional independence relationship between random variables more intuitively. Next, we will introduce the undirected graphical model, also known as is a Markov random field or a Markov network, which uses an undirected graph to describe the joint distribution of a set of random variables with Markov properties.

Definition (Markov Random Field): For a random vector $X(X_1,X_2,\cdots ,X_n)$ and there arennUndirected graphG = ( V , E ) G=(V,E)$with\; n\; nodes$$G=(V,E)$，其中$V$表示节点集合且 V = { X 1 , X 2 , ⋯   , X n } V=\{X_1,X_2,\cdots,X_n\} V={ X1​,X2​,⋯,Xn​}，$E$表示节点之间边的集合，如果$(G,X)$满足局部马尔可夫性质，即给定变量$X_k$邻居的取值，其与其他的变量独立
$$P(X_k=x_k|X_{-k})=P(X_k=x_k|X_{N(k)})$$
其中$X_{-k}$means except $X_k$External to the set of other random variables, $X_{N(k)}$means $X_k$The set of neighbors, that is, and $X_k$A set of random variables directly connected by edges, then we call it $(G,X)$ is a Markov random field.
Assume that the random vector in the probability graphical model obeys the multivariate Gaussian division$N(\mu ,\Sigma )$ ,fromΘ$Th=S^{-1}$ is the covariance matrix$The\; inverse\; matrix\; of\; \Sigma$ is called the accuracy matrix. We have the following proposition
$$i_{ij}=0\iff givenX{\_}_k(k\ne i,j),\; then{X}_{i}and{\_}_{j}is\; independent$$ , which means that the elements θ ij \theta_{ij}
in the accuracy matrix$i_{ij}$is 0, the node X i X_i in the undirected graph$X_i$and $X_j$There is no directly connected edge between them, that is, given neighbor information, $X_i$and $X_j$Conditionally independent, if $i_{ij}\ne 0$ , then it means$X_i$and $X_j$There are directly connected edges between them, so the precision matrix gives the structural information of the undirected graph and the parameter information of the distribution.
In practice, we care about how to learn the precision matrix from the data. Using the likelihood function of the precision matrix, we can get a convex optimization problem

Specifically, given $n$维高斯随机向量$Y=(Y_1,Y_2,\cdots ,Y_n)\sim N(\mu ,\Sigma ),\mu \in R^n,\Sigma \in S_{++}^n$的一组实际取值 { y 1 , y 2 , ⋯   , y n } ( y 1 = ( y 1 , y 2 , ⋯   , y n ) T ) \{y^1,y^2,\cdots,y^n\}(y^1=(y_1,y_2,\cdots,y_n)^T) { y1,y2,⋯,yn}(y1=(y1​,y2​,⋯,yn​)T ), its empirical covariance matrix can be obtained as
$$S=\frac{1}{m}\sum _{i=1}^m(y^i-\bar{y})(y^i-\bar{y}{)}^T$$
其中$\bar{y}=\frac{1}{m}{\sum }_{i=1}^m{y}^i$ is the sample mean.

Some students who have no foundation here may ask what empirical covariance is. My understanding is to use a set of data to infer what the approximate covariance matrix of the sample looks like.
About precision matrix XXThe log likelihood function of$X$
$$l(x)=lndet\left(X\right)+Tr(XS),X\succ 0whereX$$
≻$X\succ 0$ represents the independent variable$X$ takes the value in the positive definite matrix space by maximizing the log likelihood function
$$\underset{X\succ 0}{\max }l(X)\text{\hspace{1em}( 3.5.1 )}$$
We can get the accuracy matrix$Estimates$ of Not fully connected, we build the following improved model:
$$\underset{X\succ 0}{\max }l\left(X\right)-\lambda ||X|{|}_1\text{\hspace{1em}( 3.5.2 )}$$
where$l>0$ is a parameter used to control sparsity. The sparse solution obtained by the above model can be used to estimate the conditional independence between high-dimensional random variables.

In addition to using the likelihood function to model, we can also directly design the optimization problem from the idea of ​​adding a regular term to the loss function. According to the definition of the accuracy matrix, the real accuracy matrix Θ = Σ − 1 \Theta={\Sigma $Th=S^{-1}$ , Σ \Sigmacan be obtained through samplingEstimateSS$of\; \Sigma$$S$ , which is the empirical covariance matrix mentioned above, our goal is to estimate$S^{-1}$ , and make it have a sparse structure, so the following optimization problem can be designed
$$\underset{X}{\min }||SX-I||+\lambda ||X|{|}_1,s.t.X\succ 0\text{\hspace{1em}(3.5.3)}$$
其中$||\cdot ||$可以是任意一种范数（较为常用的为$F$范数或者$l_{!}$范数）$X\succ 0$表示$X$在半正定矩阵空间中取值，每一项的含义是很明显的，第一项表示希望$X$尽可能为$S$的逆矩阵，$||X|{|}_1$是要求$X$本身稀疏，$X\succ 0$是保证了求得是精度矩阵是半正定的，我们也可以给出问题的一个变形
$$\underset{X}{\min }||X|{|}_{1}s.t.||SX-I||\le \sigma ,X⪰0\text{\hspace{1em}(3.5.4)}$$
满足一定误差范围内的$X$寻找$l_1$范数最小的解，当然如果$\sigma$过小，可行域可能是空集

3.6 相位恢复

相位恢复是信号处理中的一个重要问题，它是从信号在某个变换域的幅度测量值来恢复信号。
问题背景如下：将待测物体（信号）放置在指定位置，用投射光照射，经过衍射成像，可以由探测器得到其振幅分布，我们需要从该振幅分布中恢复出原始信号的信息。由$Fraunhofer$衍射方程可知，探测器处的光场可以被观测物体的傅立叶变换很好的接近。但是因为实际中的探测器只能测量光的强度，因此我们只能得到振幅信息。

信号的相位通常包含丰富的信息，下图的第一列给初了两个图片Y和S，分别对他们做二维离散傅立叶变换$F$得到$F(Y)$和$F(S)$，由于变换后的图片$F(Y)$是复数矩阵，它可以由模长$|F(Y)|$和相位$phase(F(Y))$来表示，即
$$F(Y)=|F(Y)|\odot phase(F(Y))$$
其中$|F(Y)|$表示对每个元素取模长，运算$\odot$表示矩阵对应元素相乘，现在交换$Y$和$S$的相位，但保留模长，然后做傅立叶逆变换$F^{-1}$，即
$$\begin{gathered} \hat{S}=F^{-1}(|F(Y)|\odot phase(F(S))) \\ \hat{Y}=F^{-1}(|F(S)|\odot phase(F(Y))) \end{gathered}$$

可以看到，$\hat{S}$基本上是$S$的形状，而$\hat{Y}$基本上是$Y$的形状，这个实验告诉我们相位信息可能比模长信息更重要。
（信号处理这一块我实在是不是很熟，我只能尽量的理解）
在实际应用中，我们不一定使用傅立叶变换对原始信号进行采样处理，给定复信号$x=(x_0,x_1,\cdots ,x_{n-1}{)}^T\in C^n$以及采样数$m$,我们可以逐分量定义如下线性变换：
$$(A(x){)}_k={\bar{a}}_{k}x,k=1,2,\cdots ,m$$
其中$a_k\in C^n$为已知复向量。$\bar{a}$应该是表示复向量的共轭转置，容易验证当该线性变换$A$为离散傅立叶变换时，$a_k$有如下形式：
$$a_k=(e^{2\pi i\frac{k-1}{n}t}{)}_{t=0}^{n-1},k=\mathrm{1.2.}\cdots ,m$$
针对一般形式的$a_k$，如果将其对应的振幅观测记为$b_k$. Then the phase recovery problem is essentially to solve the following quadratic equations:
$$b_k^2=|{\bar{a}}_k^{T}x{|}^2,k=1,2,\cdots ,m\text{\hspace{1em}( 3.6.1 )}$$
How should we understand this? It is said above that$b_k$It should correspond to a transformation. It is said to be amplitude here. My understanding is that the amplitude that can be detected can be obtained by transforming the original signal. This amplitude should also have a corresponding ak $a_k$, or each different kind of signal has its own $a_k$? I don’t quite understand it either. For the time being, I can only understand it this way and can barely explain it.

Although solving a system of linear equations is very simple, solving a system of quadratic equations is an NP-hard problem. Below we introduce two methods of converting the problem into a solvable optimization model.

1. Least squares model

It is more common to transform problem (3.6.1) into a nonlinear least squares problem
$$\underset{x\in C^n}{\min }\sum _{i=1}^m(|{\bar{a}}_i^{T}x{|}^2-b_i^2{)}^2\text{\hspace{1em}( 3.6.2 )}$$
The goal of this model is a differentiable quartic function, which is a non-convex optimization problem. Compared with the problem (3.6.1), the model (3.6.2) can better deal with the noise in the observation. In practice, we often construct the following non-smooth model
$$\underset{x\in C^n}{\min }\sum _{i=1}^m(|{\bar{a}}_i^{T}x|-b_i{)}^2\text{\hspace{1em}( 3.6.3 )}$$
假设$a_i$and $All\; x$ are real, and we can get the corresponding model in the case of real numbers:
$$\underset{x\in C^n}{\min }\sum _{i=1}^m(|<{\bar{a}}_i,x>{|}^2-b_i^2{)}^2\text{\hspace{1em}(3.6.4)}$$
$$\underset{x\in C^n}{\min }\sum _{i=1}^m(|<{\bar{a}}_i,x>|-b_i{)}^2\text{\hspace{1em}(3.6.5)}$$

Because the phase recovery problem has important applications in practice, how to find the global optimal solution of models (3.6.2)-(3.6.5) has attracted widespread attention.

2. Phase improvement

Phase lift (phase lift) is another method to solve the phase recovery problem. In front of the razor, the essential difficulty of the phase recovery problem lies in dealing with the quadratic equations (3.6.1). Note that ∣ a ˉ i T x ∣ 2 =
$$|{\bar{a}}_i^{T}x{|}^2={\bar{a}}_i^{T}x\bar{x}{a}_i=Tr(x{\bar{x}}^T{a}_i{\bar{a}}_i^T)$$

令$X=x{\bar{x}}^T$ , the system of equations 3.6.1 can be transformed into
$$Tr(X{a}_i{\bar{a}}_i^T)=b_i^2,i=1,2,\cdots ,m,X⪰0,rank(X)=1\text{\hspace{1em}( 3.6.6 )}$$
If$X=x{\bar{x}}^T$ , thenXXThe rank of$X\; is\; usually\; 1.$

This is because $X$ is a column vector$x$ and its conjugate transpose${\bar{x}}^{}$obtained by multiplying${}^{T.}$The properties of this matrix are called rank-1 matrices. A rank-1 matrix always has rank 1 because all columns are linear combinations of other columns.

Specifically, if $x$ is a non-zero complex vector, then$X$ is a rank-1 matrix with rank 1. if$x$ is a zero vector, then$X$ is also a zero matrix, with rank 0.

In short, $X=x{\bar{x}}^{The\; rank\; of\; T}$ is usually 1 unless$x$ is a zero vector.

If the system of equations (3.6.1) has $x$ exists, then$X=x{\bar{x}}^T$ is the solution of the system of equations (3.6.6). For the system of equations (3.6.6), we consider the optimization problem
$$\begin{gathered} \underset{X}{\min }rank(X) \\ s.t.Tr(X{a}_i\overline{a_i}=b_i^2),i=1,2,\cdots ,m \\ X⪰0\text{\hspace{1em}( 3.6.7 )} \end{gathered}$$
Because a rank-one solution exists, the rank of the optimal solution to problem (3.6.7) is at most 1. Make a rank-one decomposition$X=x{\bar{x}}^T$。那么$cx,c\in C$ and$|c|=1$ is the solution to equation (3.6.1).
The above discussion shows that the problem (3.6.7) and the phase recovery problem (3.6.1) are equivalent, and the difference in form is that the independent variable of the problem (3.6.7) is the matrix$X$ , the operation of converting vector variables into matrix variables is also called "lifting". The purpose of using lifting is to change the constraints from about the vector$The\; quadratic\; function\; of\; x$ is transformed into matrixXXlinear function of$X.$

Because of the computational complexity of rank optimization, we use the nuclear norm to relax it, and get the following optimization problem
$$\begin{gathered} \underset{X}{\min }Tr(X) \\ s.t.Tr(X{a}_i{\overline{a_i}}^T)=b_i^2,i=1,2,\cdots ,m,X⪰0\text{\hspace{1em}( 3.6.8 )} \end{gathered}$$
where$Tr(X)=||X||$ , from matrix$The\; positive\; semi-definiteness\; of\; X$ , note that the objective function of the problem (3.6.8) becomes a linear function, and its only non-linear part is the positive semi-definite constraint$X⪰0$ , when there is a unique solution to the problem (3.6.8), the original phase recovery problem can be obtained by rank-one decomposition. Some articles have proved that when$m\ge c_{0}nlnn)$ ($c_0$is a problem-dependent constant), the solution to problem (3.6.8) is rank-one with high probability.

PS: Rank-One Decomposition is a decomposition method that represents a matrix as a linear combination of a rank-1 matrix. In linear algebra, for a matrix, the form of rank-one decomposition is usually:
$$A=u{v}^T$$
,$A$ is a matrix,$u$ sum$v$ is a vector,${\cdot }^T$ represents the transpose of a vector. The key feature of this decomposition is that$A$ can be expressed as the outer product of two vectors. Vector$u$ sum$v$ is called the basis vector of decomposition, and their outer product$u{v}^T$ is a matrix of rank 1.

3.7 Principal component analysis

Principal component analysis is an important technique for data processing and dimensionality reduction. It provides a method to express points in high-dimensional space in low-dimensional subspace. Given data ai ∈ R p , i = 1 , 2 , ⋯ , $a_i\in R^p,i=1,2,\cdots ,$nwhere$nn$$n$ represents the number of samples, define$A=[a_1,a_2,\cdots ,a_n]$ , without loss of generality, we assume thatAAThe row sum of$A\; is\; 0\; (otherwise\; the\; average\; value\; of\; the\; row\; can\; be\; subtracted,\; and\; the\; relative\; structure\; of\; the\; data\; will\; not\; change).$
The idea of ​​principal component analysis is to find a subspace composed of several directions with the largest variance of sample points, and then project data points into the subspace to achieve dimensionality reduction.
PS: In fact, there is a more easy-to-understand way of understanding the principal components here, which may be a bit more professional mathematics.
Suppose we want to change$R^{}$Set of data points in${}^p$ { ai } i = 1 n \{a_i\}_{i=1}^n{ ai​}i=1n​, projected onto $R^{}$add${}^{of\; p}$$d-$ dimensional subspace ($d<p$)中，记$X\in R^{p\ast d}$ is the column orthogonal matrix formed by the standard orthonormal basis of the subspace

PS:(“记$X\in R^{p\ast d}$ is the column orthogonal matrix formed by the standard orthonormal basis of this subspace": In this low-dimensional subspace, we can use a matrix$X$ represents the orthonormal basis of this subspace,$X$ is a$p\times matrix\; of\; d$ , where$p$ is the dimension of the original data,$d$ is the dimension of the target subspace. This matrix$The\; columns\; of\; X$ are orthogonal, and they form a set of basis for the subspace. This matrix$X$ is often called a "projection matrix" and can be used to project original data points into a low-dimensional subspace. )

Number of points $a_i$in XXThe projection of the subspace formed by$X$$P_X(a_i)=X{X}^T{a}_i$, I still don’t quite understand it until this point. I still don’t understand it after thinking about it for a long time. $X{X}^T$ is$p\times$matrix of$p$$a_i$is $p\times 1$ (or$1\ast p$ ?), I don’t understand how he projected to$It\text{'}s\; d-$ dimensional. If anyone sees this and can give me an answer, thank you! !

So let me talk about PCA that I understood before.

First of all, if we want to describe a vector, we need a set of bases. It is enough to give the projection values ​​of the vector on the straight line where the base is located. If the two-dimensional vector wants to be reduced to one-dimensional, find a set of bases. As shown below, we can project all these two-dimensional data points to $x_1$superior

The question is, what do we need to satisfy this base? If we want to retain more information, we must make the data dispersed as much as possible. So how to express data dispersion? That's right! Variance
We can transform the above problem into finding a set of basis, so that when all the data is transformed to this set of basis, the variance is maximum

For the problem of reducing two dimensions to one dimension, we only need to find the direction that maximizes the variance, but for higher dimensions, there is another problem to be solved. If you reduce three dimensions to two dimensions, first we hope to find One direction maximizes the variance, thus completing the selection of the first direction, and then we select the second direction.
If we simply choose the direction with the largest variance at this time, it is obvious that this direction will almost coincide with the first direction. Obviously, such a dimension is useless. Intuitively speaking, these two directions should be as close as possible To represent more information, we don't want a linear correlation between them, because correlation means that the two directions will definitely express the same information repeatedly.

Mathematically, the covariance of two fields (attributes) represents their correlation. As we have said before, let the mean value of each field (attribute) be 0, then cov ( a , b ) = 1 m ∑ i = 1
$$cov(a,b)=\frac{1}{m}\sum _{i=1}^m{a}_i{b}_i$$

Everything mentioned now is the prerequisite, and we haven’t reached the PCA part yet.
Suppose we only $a$ and$b$ two attributes, we form them into a matrix$X$
$$X=\left(\begin{array}{cccc}{a}_1& a_2& \cdots & a_n\\ b_1& b_2& \cdots & b_n\end{array}\right)$$
we use$X$ times$X^T$,并乘上系数1\m：
$$\frac{1}{m}X{X}^T=\left(\begin{array}{cc}\frac{1}{m}{\sum }_{i=1}^m{a}_i^2& \frac{1}{m}{\sum }_{i=1}^m{a}_i{b}_i\\ \frac{1}{m}{\sum }_{i=1}^m{a}_i{b}_i& \frac{1}{m}{\sum }_{i=1}^m{b}_i^2\end{array}\right)$$
A miracle happened! , this is called the covariance matrix. The two elements on the diagonal of this matrix are the variances of the two attributes, while the other elements are$a$ andbbCovariance of$b$
So what do we hope now? We hope that the transformed data is assumed to be$Y$ ,empty
$$Y{Y}^T=\left(\begin{array}{cccc}a& 0& 0& 0\\ 0& b& 0& 0\\ 0& 0& c& 0\\ 0& 0& 0& d\end{array}\right)$$
Is it a square array similar to this? It's similar. The specific size is uncertain. I just hope that there are only elements on the diagonal and all other elements are 0.

Let the original data matrix $X\in R^{}$The covariance matrix corresponding to${}^p$${}^{\ast }$${}^n$$P\in R^d\ast p$ is a matrix composed of a set of bases (after dimensionality reduction) by row. Suppose$Y=PX\in R^{d\ast n}$ ,则$Y$ is$X$ versus$P$ is used as the basis transformed data, assumingYYThe covariance matrix of$Y$$D$ , let us derive$D$ and$C$的关系
$$D=\frac{1}{m}Y{Y}^T=\frac{1}{m}(PX)(PX{)}^T=\frac{1}{m}PX{X}^T{P}^T=P(\frac{1}{m}X{X}^T)P^T=PC{P}^{TNow}$$
we understand,$P$ is the PPthat diagonalizes the original covariance matrix$P.$ _ In other words, the optimization goal becomes to find a matrix$P$ , satisfies$PC{P}^T$ is a diagonal matrix, and the diagonal elements are arranged in order from large to small, the first$Line\; K$ means we want to drop to$K-$ dimensional basis, therefore, we should thank mathematicians for their advance.

It is known from the above that the covariance matrix is ​​a symmetric matrix. In linear algebra, the real symmetric matrix has a series of very good properties
(1) The eigenvectors corresponding to different eigenvalues ​​​​of the real symmetric matrix must be orthogonal
(2) Let the eigenmatrix $\lambda$ multiplicity is$r$ , then there must be$r$ linearly independent eigenvectors corresponding to$\lambda$ , so this$Orthogonalization\; of\; r$ eigenvector units
From the above, it can be seen that an$n$ lines$A\; real\; symmetric\; matrix\; with\; n$ columns must be able to find$n$ unit orthogonal eigenvectors, let this$The\; n$ feature vectors are$e_1,e_2,\cdots ,e_n$，记
$$E=(e_{!},e_2,\cdots ,e_n)$$
对协方差矩阵有如下结论：
$$E^{T}CE=A=\left[\begin{array}{cccc}{\lambda }_{!}& 0& \cdots & 0\\ 0& {\lambda }_2& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & {\lambda }_n\end{array}\right]$$
那么我们已经找到我们需要的矩阵$P$了，$P=E^T$
$P$是协方差矩阵的特征向量单位化后按行排列出的矩阵，其中每一行都是$C$的一个特征向量，如果设$P$ according to$The\; eigenvalues\; of\; A$ are from large to small, arrange the eigenvectors from top to bottom, and use the firstKKThe matrix consisting of$K$$X$ , you can get the dimensionally reduced data matrix$Y$ is
here. I have doubts here. The directions selected by these eigenvalues, in addition to ensuring that the directions are orthogonal, can only guarantee the relative size? There is no guarantee that his direction has the largest variance among all directions, and then I searched. There are mathematical derivations that can prove that the eigenvector is the direction that can maximize the variance.
Let’s borrow this article from Zhihu: https://zhuanlan.zhihu.com/p/338322144

To summarize: Suppose there are $m$ n$n-$ dimensional data
(1) compose the original data into$n$ lines$m-$ column matrix$X$
(2) willXX
Eachrow$of$$C=\frac{1}{m}X{X}^T$
(4) Find the eigenvalues ​​of the covariance matrix and the corresponding eigenvectors
(5) Arrange the eigenvalues ​​corresponding to the eigenvectors into a matrix from top to bottom, and take the first$K$ rows make up the matrix$P$
（6）$Y=PX$ is dimensionality reduction tokkData after$k\; dimension$

3.8 Matrix separation problem

The matrix separation problem is also an important low-rank matrix calculation problem. Given the matrix $M\in R^{m\times n}$ , we want to decompose it into a low-rank matrix$X$ and sparse matrix$S$ , such that$X+S=M$ , while trying to make the matrixXXrank sum sparse matrixSS$of\; X$$S$ l$l_0$The norms are relatively small, so the following model
$$\begin{gathered} \underset{X,S\in R^{m\times n}}{\min }rank(X)+\mu ||S|{|}_0 \\ s.t.X+S=M\text{\hspace{1em}( 3.8.1 )} \end{gathered}$$
Since the model contains the rank sum of the matrix$l_0$Norm is difficult to solve directly. So we use the nuclear norm instead of rank and minimize $l_1$The norm restricts the sparsity of the noise matrix, (kernel norm: the singular value addition of the matrix)
$$\begin{gathered} \underset{X,S\in R^{m\times n}}{\min }||X|{|}_{\ast }+\mu ||S|{|}_1 \\ s.t.X+S=M\text{\hspace{1em}( 3.8.2 )} \end{gathered}$$
The matrix separation problem is sometimes called robust principal component analysis (robust PCA), the goal is to remove the noise in the original data to the greatest extent in image processing, and find the optimal projection.

Limitations Consider the video segmentation problem, which refers to extracting objects of interest from video scenes. For example, segment a still part of a video. Each frame of the video is actually a static picture. Although the static objects in each picture may be affected by lighting changes, occlusion, translation, noise, etc., resulting in subtle differences between different pictures, it is undeniable that they are different from each other. There is a high degree of similarity between them. If the static parts in all pictures are represented as a matrix, and because the static objects have a certain internal structure, the matrix composed of static objects must be of low rank (each row or column is linearly related), similarly, the dynamics in the video part and other background factors can be regarded as noise, then our task becomes to decompose the information matrix contained in the video into the sum of low-rank matrix and sparse noise matrix with internal structure.

We choose actual video data to illustrate specifically, assuming that the video has a total of $n$ frames, each frame (each picture) has a total of$m$ pixels,we represent each picture as a column vector, and these column vectors are put together to form the given matrix$M$ , since the background of the lobby is basically the same in every frame, it corresponds to$The\; low-rank\; part\; of\; M$ , let$m=\frac{1}{2\sqrt{m}}$Solving the model (3.8.2), we can get the matrix $X$ and$S$ respectively opposite the second and third columns

3.9 Dictionary learning

Just as all kinds of knowledge can be expressed by permutations and combinations of words in a dictionary, the purpose of dictionary learning is to compress existing large-scale data sets and find the most basic knowledge hidden behind these data points. principle.
consider a mmData set aii = 1 n in$m-$${a_i}_{i=1}^n,a_i\in R^m$ , assuming that each$a_i$are all generated by the same dictionary, and the generated data contains noise, so the linear model of dictionary learning can be expressed as
$$a=Dx+ehereD$$ ∈ R m × k D{\in}R^{m \times k
}$D\in R^{m\times k}$ is an unknown dictionary, and each of its columns$d_i$is a basis vector of the dictionary; $x$ is the coefficient of the basis in the dictionary, also unknown;$e$ is some kind of noise. How to understand here, let's imagine, suppose you have 1,000,000 pictures, each picture is 1010, that is, 100 dimensions, and now through dictionary learning, you can find$k$ pictures of 100 dimensions (1010), used as a dictionary, through this$k$ pictures to represent these 1,000,000 pictures.
The dictionary learning model is different from the multiple linear regression model because we need to solve the dictionary$D$ and coefficient$x$ (each picture has its own corresponding$x$)。

Generally speaking, the dimension of the data $m$ and the number of basis vectors in the dictionary$k$ is much smaller than the number of observations$n$ if$k<m$ , we call the dictionary$D$ is incomplete if$k>m$ , we call the dictionary$D$ is overcomplete,$k=The\; dictionary\; corresponding\; to\; m$ cannot bring any improvement to the representation and is not considered in practice. (Over-complete dictionaries are generally$x$ are all sparse, which also has advantages)

This eeWhen$e$
$$f(D,X)=\frac{1}{2n\_}||DX-A|{|}_F^2$$
其中 $A=[a_1,a_2,\cdots ,a_n]\in R^{m\times n}$ is the total of all observation data,$X=[x_1,x_2,\cdots ,x_n]\in R^{k\times n}$ is the total set of all basis coefficients
. In actual calculations, we do not require$The\; columns\; of\; D$ are orthogonal (meaning that the basis vectors are not required to be linearly independent), so a sample point$a_i$There may be many different representations, and this redundancy introduces sparsity into the representation, which means that the dictionary is overcomplete. Sparsity can also help us quickly determine which basis vectors represent sample points, thereby improving calculation speed. Specifically, in eeUnder the condition that$e$
$$f(D,X)=\frac{1}{2n\_}||DX-A|{|}_F^2+\lambda ||X|{|}_1$$
where $\lambda$ is a regularization parameter, its size is used to controlXXThe sparsity of$X$$f(D,X)$ has a product term$DX$ , obviouslyffThe minimum value point of$f$$||X|{|}_1\to 0$ (because assuming$(D,X)$ is the minimum point of the problem, then$f(cD,\frac{1}{c}X)<f(D,X),\forall c>1$ ), this should be easy to understand,$DX$ product will not change, but$As\; X$ becomes smaller, the regular term becomes smaller, resulting in$f(D,X)$ becomes smaller.
Therefore, the sparse-preserving regular term here is meaningless. An improved approach is to require that the base vector module length in the dictionary cannot be too large.
The final optimization problem is
$$\begin{gathered} \underset{D,X}{\min }||DX-A|{|}_F^2+\lambda ||X|{|}_1 \\ s.t.||D|{|}_F\le 1\text{\hspace{1em}( 3.9.1 )} \end{gathered}$$
In this case, it is impossible to alwaysc D cDSuch a change in$cD\; limits\; D\; from\; being\; too\; large,\; so\; X\; will\; not\; be\; too\; small.$