Linear Programming and Simplex Method - Principles

introduction

Many operational research textbooks start with linear programming. When I usually do the application of algorithm strategies, I also develop some technical solutions based on linear programming. It can be said that if you do not understand linear programming, it is difficult to become an excellent operational research optimization algorithm engineer.

But when I was studying systematically, I made a big circle in other places before I came here.

The main reason is that the principle of this linear programming is really a bit difficult. I read it many times before, and there is always a sense of frustration that I seem to understand but not fully understand. After learning from the pain, I finally decided to start with the simplest unconstrained problem, and then gradually transition to the constrained problem, and the current linear programming problem.

For unconstrained optimization problems, I subdivided them into one-dimensional problems and multi-dimensional problems, and learned the golden section method , tangent method and advance and retreat method suitable for solving one-dimensional problems, as well as the coordinate rotation method and the fastest method for solving multi-dimensional problems. Descent method and quasi-Newton method , etc.; for constrained optimization problems, Lagrange multiplier method and penalty function method have been learned successively .

The linear programming problem is essentially a special kind of constrained optimization problem. This kind of optimization problem is very common in practical scenarios. At the same time, due to the particularity of the problem, it also brings new ideas to the efficient solution of the problem.

As mentioned earlier, linear programming is very difficult for me, and it is difficult to explain clearly in only one article. So I plan to divide it into two parts: this one focuses on explaining the principle of the algorithm, and the next one focuses on the practical application.

It must be warned that the follow-up content is very boring, because there is a lot of proof and reasoning process mixed in. I am willing to spend a lot of time on this, hoping not only to know what it is, but also to know why it is. If you don’t want to look at the detailed process, you can pay more attention to the overall logical framework: where is the particularity of the linear programming problem compared to the general constraint problem –> what characteristics does this particularity make the linear programming problem have –> based on these characteristics, the designed algorithm (Simple method) is how to achieve efficient solution.

standard form of linear programming

The standard matrix form of linear programming is
$$\begin{gathered} minf(x)=c^{T}x \\ \text{s.t.}Ax=b \\ x\ge 0 \end{gathered}$$

It can also be written in the component form
$$\begin{gathered} minf\left(x\right)=\sum _{j=1}^n{c}_j{x}_j \\ \text{s.t.}\sum _{j=1}^n{a}_{ij}{x}_j=b_i,i=1,2,...,m \\ {x}_j\ge 0,j=1,2,...,n \end{gathered}$$

It can be seen from the definition that compared with general constrained optimization problems, linear programming problems have added restrictions on the objective function, constraints and variable range: (1) the objective function is an expression of linear summation; (2) the constraint The condition is only linear equality; (3) The optimization variable is greater than or equal to 0.

Problem characteristics

What are the characteristics of the linear programming problem after adding the above restrictions?

Let me talk about the conclusion first: the feasible domain of the problem is a convex set –> the optimal solution is on the vertex of the convex set –> the vertices correspond to the basic feasible solutions.

Next, explain in detail (proof) one by one.

(1) The feasible region is a convex set.

Now that convex sets are mentioned, it is necessary to define the meaning of convex sets first: Let $S\in R^n$ is$A\; point\; set\; in\; n-$ dimensional Euclidean space, if for$x_1\in S,x_2\in S,x_1\ne x_2$, and any $l\in [0,1]$，必有
$$\lambda x_1+(1-l)x_2\in S$$
is called$S$ is a convex set.

The following are examples of convex and non-convex sets

For linear programming problems, according to the definition of standard constraints

$$\begin{gathered} A{x}_1=b,x_1\ge 0 \\ A{x}_2=b,x_2\ge 0 \end{gathered}$$
Then, for any$l\in [0,1]$ , suppose the new variable$x=\lambda x_1+(1-l)x_2$，可做方法推密
$$A[\lambda x_1+(1-l)x_2]=\lambda A{x}_1+(1-\lambda )A{x}_2=\lambda b+(1-l)b=b$$
means that the equality constraints are established.

Since $x_1,x_2,l\ge 0$ , the following formula is obviously also true
$$\lambda x_1+(1-l)x_2\ge 0$$
is$It\; is\; also\; true\; that\; x$ is greater than or equal to 0.

so $x$ is also in the feasible region. According to the definition of a convex set, the feasible region of a linear programming problem is a convex set.

The feasible region is a convex set, what are the benefits - see the next conclusion.

(2) The optimal value must be reached at a certain vertex of the convex set.

The word vertex is easy to understand in the physical world, but for subsequent derivation, it is still necessary to describe this concept clearly in mathematical language: for x \pmb x in the feasible region$x$ , if there is no$l,x_1\in S,x_2\in S,x_1\ne x_2$, making
$$x=\lambda x_1+(1-l)x_2$$
Then this $x$ is a vertex on a convex set.

To prove the above conclusion, the method of proof by contradiction is used here.

Suppose the vertices of the convex set are $x_1,x_2,...,x_k$, the optimal solution is $x^{\ast }$ . Because$x^{\ast }$ is within the convex set of the feasible region, so it can be represented by vertices as
$$x^{\ast }=\sum _{i=1}^k{l}_i{x}_i$$
where $l_i\ge 0$，${\sum }_{i=1}^k{l}_i=1$。$It\; is\; easy\; to\; imagine\; that\; x$ can be represented by a linear combination of vertices, but it is a bit difficult to prove, so I will not go into depth here. Those who are interested can refer to: the representationtheorem of multi-faceted sets.

Multiply both sides of the above formula at the same time by $c^T$
$$c^T{x}^{\ast }=\sum _{i=1}^k{l}_i{c}^T{x}_i$$
设$c^T{x}_s=\underset{1\le i\le k}{\min }{c}^T{x}_i$, the above formula can be deduced as follows

$$c^T{x}^{\ast }=\sum _{i=1}^k{l}_i{c}^T{x}_i\ge \sum _{i=1}^k{l}_i{c}^T{x}_s=c^T{x}_s\sum _{i=1}^k{l}_i=c^T{x}_s$$
But since $x^{\ast }$ is the optimal solution and is not on the vertex, so
$$c^T{x}^{\ast }<c^T{x}_s$$
The above two formulas contradict each other, so the assumption is not true, that is, the optimal solution must be obtained at a certain vertex.

The biggest advantage of this conclusion is that it shrinks the decision space from a very large feasible domain space to the vertices of the feasible domain, and the number of vertices is relatively limited in most cases, which greatly reduces the complexity of problem solving Spend.

In fact, I still have a small doubt here. From the perspective of the proof process, it seems that the objective function does not need to be linear. Does it mean that even if the objective function is a nonlinear expression, the optimal solution is still obtained at the vertex? This doubt has not been clarified yet, if there is a big brother, I hope you can enlighten me.

(3) There is a one-to-one correspondence between the vertices of the convex set in the feasible region and the basic feasible solutions.

We already know that the optimal solution is on the vertex, so what is the mathematical expression of the vertex? The above conclusion tells us that the vertex is the basic feasible solution.

Next, let's look at the definition of a basic feasible solution.

$A$是$m\times The\; constraint\; matrix\; of\; n$ , from which a non-singular square matrix B \pmb Bcan be taken$B$ , and then make the remaining columns into a sub-array$N$ , their corresponding$x$ are represented as basic variables$x_B$and non-basic variable $x_N$. Equality constraints can be expressed as
$$B{x}_b+N{x}_N=bLet$$
x N = 0$x_N=0$ , we can find$x_b=B^{-1}b$, combination$x_b$and $x_N$
$$x=(B^{-1}b,0{)}^{TIf}$$ B − 1 b ≥ 0 \pmb B^{-1}\pmb
b≥0$B^{-1}b\ge 0$ , then$x$ is called the basic feasible solution.

$The\; dimension\; of\; B$ is$m\times m$ , theoretically at most from anynnRandomly selectmm$from\; n\; columns$$m$ columns, so$The\; upper\; limit\; of\; the\; number\; of\; B$ is$C_n^m$, that is, the upper limit of the number of basic feasible solutions is $C_n^m$indivual.

The above conclusion needs to be demonstrated in two steps: (1) the basic feasible solutions are vertices; (2) the vertices are all basic feasible solutions.

For the first step, continue to use the method of proof by contradiction.

Suppose a basic feasible solution is not a vertex, that is, there is a feasible solution $x_1,x_2$sum $l\in [0,1]$ , making the basic feasible solution$x$ can be expressed as
$$x=\lambda x_1+(1-l)x_2$$
put $x$ is written as the base variable$x_B$and non-basic variable $x_N$Two parts
$$x_B=\lambda x_{1B}+(1-l)x_2$$
$$x_N=\lambda x_{1N}+(1-l)x_2$$
$x_N$In the equation, because $x_N=0,l>0,1-l>0,x_{1N}\ge 0,x_2\ge 0$，故
$$x_N=x_{1N}=x_2=0$$
so$x_1$and $x_2$are basically feasible solutions, that is,
$$B{x}_{1B}=b$$
$$B{x}_2=b$$
due to$B$ is a non-singular square matrix, so$Bx=b$ has a unique solution, so
$$x_B=x_{1B}=x_2$$
That is, $x_1,x_2$is the basic feasible solution $x$ , does not match the basic assumption, so the basic feasible solution must be the vertex of the feasible solution.

For the second step, the method of proof by contradiction is still used.

Suppose $Is\; x$ the vertex of the feasible region, or is it first disassembled into non-zero items$x_B$and zero term $x_N$
$$B{x}_B=b$$
but because it is not a basic feasible solution, it corresponds to$B$ should be linearly related, that is, there is a set of non-zero vectors$w$ , making
$$Bw=0$$
The above formula is multiplied by the coefficient$\delta$ , and then do addition and subtraction with the above formula respectively to get
$$\begin{gathered} B(x_B+\delta w)=b \\ B(x_B-\delta w)=b \end{gathered}$$
先令$x_B^1=x_B+\delta w,x_B^2=x_B-\delta w$，再令$x_1=(x_B^1,0),x_2=(x_B^2,0)$ , first we can get
$$x=\frac{1}{2}{x}_1+\frac{1}{2}{x}_2$$
At the same time as long as $\delta$ is small enough
$$x\pm \delta w>=0$$
is$x_1$and $x_2$are all feasible solutions, which means that $x$ cannot be a vertex, and the two are contradictory, so the vertices must also be basic feasible solutions.

With this conclusion, in theory, as long as the objective function values ​​of all basic feasible solutions are calculated and the minimum value is returned, the optimal solution is obtained. But the upper limit of the number of basic feasible solutions is $C_n^m$, if $The\; number\; of\; n$ is relatively large, and it seems a bit overwhelming to use traversal to find the minimum value.

At this time, the famous simplex method should be on the stage.

simplex method

Also give the conclusion first, the most powerful part of the simplex method is: after arbitrarily given a basic feasible solution, after simple calculation and evaluation, it can tell us whether the solution has room for improvement, and if so, in which direction Improvement is best. In this case, it is no longer necessary to traverse all the basic feasible solutions, so the efficiency of problem solving can be greatly improved.

Suppose a basic feasible solution
$$x_0=[x_B,0]=[B^{-1}b,0]$$
The corresponding objective function value is
$$f_0=c_B^T{B}^{-1}b$$

First put $x_0$General expression written as a basic feasible solution $[x_B,x_N]$ , substitute into the equality constraint
$$B{x}_B+N{x}_N=b$$
transposition,$x_B$Can be expressed as
$$x_B=B^{-1}b-B^{-1}N{x}_N$$

After the general expression is combined with the above formula, then substitute into the objective function expression
$$f=c_B^T(B^{-1}b-B^{-1}N{x}_N)+c_N^T{x}_N$$
Reorganize
$$f=c_B^T{B}^{-1}b-(c_B^T{B}^{-1}N-c_N^T)x_N$$
The above formula can be described as
$$f=f_0-\sum _{j\in R}(z_j-c_j)x_j$$
Among them, $R$ is a set of non-basic variables,$z=c_B^T{B}^{-1}N$。

At current $x_0$, since $x_j=0$ , ie$The\; last\; item\; of\; f$ is 0; so the basic condition for the objective function to continue to decrease is: there is at least one$j$，使得$z_j-c_j>0$ , while$x_j$It can also increase from 0 to a positive number.

$z_j-c_j$It is a parameter not a variable, whether there is $j$ , cannot be changed; but$x_j$The value of can be optimized in theory, and of course the value cannot be increased arbitrarily. The constraint that needs to be satisfied is
$$x_B=B^{-1}b-B^{-1}N{x}_N\ge 0$$

Suppose $x_k\in x_N$, will change from 0 to a positive number (the technical term is called basis), in order to ensure the basic feasible solution requirements, $x_B$There needs to be a value in 0 (the technical term is called the basis)
$$x_B=\left[\begin{array}{c}{x}_B\\ x_{B2}\\ \cdot \cdot \cdot \\ x_{Bm}\end{array}\right]=\left[\begin{array}{c}{\bar{b}}_1\\ {\bar{b}}_2\\ \cdot \cdot \cdot \\ {\bar{b}}_m\end{array}\right]-\left[\begin{array}{c}{y}_1\\ y_2\\ \cdot \cdot \cdot \\ y_{mk}\end{array}\right]x_k\ge 0$$
Among them,$\bar{b}=B^{-1}b$，$y_k=B^{-1}{N}_k$. In order to guarantee $x_B\ge 0$，$x_k$The optimal value of is
xk = min { b ˉ 1 y 1 k , b ˉ 2 y 2 k , . . . , b ˉ mymk } x_k=min\{\frac{\bar b_1}{y_{1k}}, \frac{\bar b_2}{y_{2k}},...,\frac{\bar b_m}{y_{mk}}\}xk​=min{ y1 kbˉ1​​,y2 kbˉ2​​,...,ymk​bˉm​​}

Based on the above logic, we can describe it as: if there is $k$ , such that$z_k-c_k>0$ , at this time$x_k=min\frac{{\bar{b}}_i}{y_{}}$, which can reduce the value of the objective function to the greatest extent.

Here are some special cases that need to be considered separately:

(1) Possession $z_i-c_i\le 0$ , ie$f\le f_0$, the current basic feasible solution is the optimal solution.

(2) $x_k=min\frac{{\bar{b}}_i}{y_{}}$中，${\bar{b}}_i$is the component of the original basic feasible solution, so its value ≥ 0 is beyond doubt; but $y_{}$The size of is indeterminate if its partial value ≤ 0, combined with $x_k\ge 0$ self-constraint, the original process can continue; but if all ≤ 0, the process can no longer continue, in fact, even if$x_k$Take infinity, $x_B=\bar{b}-y_k{x}_k\ge 0$ constraint can still be satisfied,$f$ will become infinitely small, so the original problem has no lower bound at this time, and there is no minimum value.

In summary, the basic steps of the simplex method can be summarized as follows:

(1) Transform the given linear programming problem into a standard form;

(2) Find an initial basic feasible solution.

(3) 检验$z_i-c_i$和$\frac{{\bar{b}}_i}{y_{}}$, judge the state of the current solution: the optimal solution or no optimal solution, exit; otherwise continue.

(4) Find the best $k$ , update$x_B$, get a new basic feasible solution, go to (3).

Well, I finally figured out the algorithm principle for solving linear programming problems.