Linear programming dual problems: theoretical derivation and practical applications

Dual problem example

I have seen the word "dual" in many places before, and I always felt that this word was very high-end. The Baidu Encyclopedia of Dual Theory even writes: "The most important discovery in the early development of linear programming was the dual problem." Therefore, now that we have reached linear programming, the dual problem is naturally worth studying in depth.

I have a general understanding of the introduction of dual problems on the Internet, which can be roughly divided into two categories: one is to directly introduce the dual problem of linear programming, and the other is to start from the Lagrangian dual problem, and then express linear programming as a special The optimization problem has special properties. Since this period of time is all about linear programming, I personally feel that the first category is more suitable for me.

Let’s first give an example to intuitively understand the dual problem.

Furniture Company A produces desks, dining tables, and chairs. The production of each type of furniture requires wood and two types of skilled labor: polishing and carpentry. The quantities of various resources required to craft each type of furniture, the amount of available resources, and the selling price of each type of furniture are shown in the table below.

resource	desk	dining table	Chair	Amount of available resources
Timber (cubic meters)	8	6	1	48
Polishing time (hours)	4	2	1.5	20
Carpentry time (hours)	2	1.5	0.5	8
Selling price (yuan)	600	300	200

Find out how the company should arrange production to maximize revenue.

The solution of this problem is not difficult. Let the output of desks be $x_1$, the output of the dining table is $x_2$, the output of chairs is $x_3$, at this time the model can be described as
$$\begin{gathered} maxf=600x_1+300x_2+200x_3 \\ \text{s.t}8x{\_}_1+6x{\_}_2+x_3\le 48 \\ 4x{\_}_1+2x{\_}_2+1.5x{\_}_3\le 20 \\ 2x{\_}_1+1.5x{\_}_2+0.5x{\_}_3\le 8 \\ {x}_1,x_2,x_3\ge 0,And\; take\; an\; integer \end{gathered}$$
Obviously this problem is an integer programming problem. We use ortools to solve it. The code is as follows

from ortools.linear_solver import pywraplp


if __name__ == '__main__':
    # 声明ortools求解器，使用SCIP算法
    solver = pywraplp.Solver.CreateSolver('SCIP')

    # 优化变量，连续值
    x1 = solver.NumVar(0, solver.infinity(), 'x1')
    x2 = solver.NumVar(0, solver.infinity(), 'x2')
    x3 = solver.NumVar(0, solver.infinity(), 'x3')

    # 目标函数
    solver.Maximize(600 * x1 + 300 * x2 + 200 * x3)
    # 约束条件
    solver.Add(8 * x1 + 6 * x2 + x3 <= 48)
    solver.Add(4 * x1 + 2 * x2 + 1.5 * x3 <= 20)
    solver.Add(2 * x1 + 1.5 * x2 + 0.5 * x3 <= 8)

    # 模型求解
    status = solver.Solve()

    # 模型求解成功, 打印结果
    if status == pywraplp.Solver.OPTIMAL:
        # 变量最优解
        print('x1: {}, x2: {}, x3: {}'.format(x1.solution_value(), x2.solution_value(), x3.solution_value()))

        # 最优目标函数值
        print('best_f =', solver.Objective().Value())

    else:
        print('not converge.')

After running the code, the optimal solution can be obtained as follows

x1: 1.9999999999999996, x2: 0.0, x3: 8.0
best_f = 2800.0

It should be noted that the problem itself is an integer programming, but the code does not restrict the variables to only take integers. However, since the optimal solution to this linear programming problem is also an integer, the results are not affected. This is mainly done to facilitate subsequent comparison with dual problems.

So what is the duality problem? Dual questions can be understood from another perspective.

The new problem is described as follows: Suppose Company B wants to purchase all the resources of Company A. At this time, it needs to determine the unit price of each resource. Set the unit price of wood purchased to $w_1$(yuan/cubic meter), the unit price of polishing time is $w_2$(yuan/hour), the unit price of carpentry time is $w_3$(yuan/hour).

On the one hand, B naturally hopes that the lower the total price, the better, so the objective function is
$$minz=48w_1+20w_2+8w_3$$
On the other hand, the condition for A's willingness to sell its own resources is that the selling price cannot be lower than the profit achieved when the same amount of resources is produced by its own organization. For the desk, B needs to pay at least 600 yuan to buy 8 cubic meters of wood, 4 hours of polishing time and 2 hours of carpentry time. This constraint can be described as
$$8w_1+4w_2+2w{\_}_3\ge 600Through$$
similar reasoning, we can also get two other constraints
$$6w_1+2w{\_}_2+1.5w_3\ge 300,w_1+1.5w_2+0.5w_3\ge 200$$

Obviously, the unit price cannot be negative, so the constraints are
$$w_1,w_2,w_3\ge 0$$
In summary, a new linear programming problem is obtained as follows
$$\begin{gathered} minz=48w_1+20w_2+8w_3 \\ \text{s.t}8w_1+4w_2+2w{\_}_3\ge 600 \\ 6w_1+2w{\_}_2+1.5w_3\ge 300 \\ {w}_1+1.5w_2+0.5w_3\ge 200 \\ {w}_1,w_2,w_3\ge 0 \end{gathered}$$
The above linear programming problem is the dual problem of the original problem.

To solve the dual problem, the code is as follows

from ortools.linear_solver import pywraplp


if __name__ == '__main__':
    # 声明ortools求解器，使用SCIP算法
    solver = pywraplp.Solver.CreateSolver('SCIP')

    # 优化变量，连续值
    x1 = solver.NumVar(0, solver.infinity(), 'x1')
    x2 = solver.NumVar(0, solver.infinity(), 'x2')
    x3 = solver.NumVar(0, solver.infinity(), 'x3')

    # 目标函数
    solver.Minimize(48 * x1 + 20 * x2 + 8 * x3)
    # 约束条件
    solver.Add(8 * x1 + 4 * x2 + 2 * x3 >= 600)
    solver.Add(6 * x1 + 2 * x2 + 1.5 * x3 >= 300)
    solver.Add(1 * x1 + 1.5 * x2 + 0.5 * x3 >= 200)

    # 模型求解
    status = solver.Solve()

    # 模型求解成功, 打印结果
    if status == pywraplp.Solver.OPTIMAL:
        # 变量最优解
        print('x1: {}, x2: {}, x3: {}'.format(x1.solution_value(), x2.solution_value(), x3.solution_value()))

        # 最优目标函数值
        print('best_f =', solver.Objective().Value())

    else:
        print('not converge.')

Running the code, you can get the optimal solution as follows

x1: 0.0, x2: 100.00000000000003, x3: 99.99999999999993
best_f = 2799.9999999999995

Comparing the optimal solutions of the original problem and the dual problem, we found an interesting phenomenon
$$f^{\ast }=z^{\ast }$$
That is, the optimal objective function values ​​of the original problem and the dual problem are equal. In fact, this is not a special case, it is just one of the properties of the dual problem of linear programming.

Dual problem definition and properties

definition

The examples in the previous chapter gave us some intuitive understanding of the dual problem. This section uses some partial theoretical derivation to study the relationship and properties between the dual problem and the original problem in linear programming.

Review the standard form of linear programming
$$\begin{gathered} minf(x)=c^{T}x \\ \text{s.t}Ax=b \\ x\ge 0 \end{gathered}$$

When deriving the simplex method, there is the following formula
$$f=c_B^T{B}^{-1}b-(c_B^T{B}^{-1}N-c_N^T)x_N$$
Definition
$$R=c_B^T{B}^{-1}N-c_N^T$$
If the linear programming has reached the optimal solution, then $R\le 0$ . At this time, let$w^T=c_B^T{B}^{-1}$ , the above formula can be written as
$$w^{T}N\le c_N^T$$
Take another look at $w^{T}B$项
$$w^{T}B=c_B^T{B}^{-1}B=c_B^T$$
Because $A=[B,N]$ , combining the above two formulas, we can get
$$w^{T}A\le c^{}$$
Add a transpose to$${}^T$$
$$A^{T}w\le c$$

Let’s look at another set of derivation
$$w^{T}b=w^{T}Ax\le c^{T}x$$

Based on the above two formulas, if we put $As\; w$ is a variable, a new optimization problem
$$\begin{gathered} max{b}^{T}w \\ \text{s.t}{A}^{T}w\le c \end{gathered}$$

This optimization problem is the dual problem of the original linear programming problem. Compared with the original question, $A,b$ and$c$ is still there, just changed its position; at the same time,$x\ge 0butw$ \$w$ has no constraints.

nature

Since the dual problem can be derived from the original problem, there are naturally some interrelationships between them.

Personally, the most basic relationship should be the weak duality theorem: if $x$ is a feasible solution to the original problem,$w$ is a feasible solution to the dual problem, then$c^{T}x\ge b^{T}w$。

This theorem has actually been proven from the previous deduction. Here it is described again through the following process:
$$c^{T}x=x^{T}c\ge x^T{A}^{T}w=b^{T}w$$

An inference can be obtained based on the weak duality theorem: if $x^{\ast }$ is a feasible solution to the original problem,$w^{\ast }$ is a feasible solution to the dual problem, and we have$c^T{x}^{\ast }=b^T{w}^{\ast }$ , then$x^{\ast }$ is the optimal solution to the original problem,$w^{\ast }$ is the optimal solution to the dual problem.

From the weak duality theorem, we know that
$$c^{T}x\ge b^T{w}^{\ast }$$
While
$$c^T{x}^{\ast }=b^T{w}^{\ast }$$
So$x^{\ast }$ is the optimal solution to the original problem.

In the same way, it can be proved that $w^{\ast }$ is the optimal solution to the dual problem.

Finally, there is the most important strong duality theorem in practical applications: if the original problem has an optimal solution, then the dual problem also has an optimal solution, and the objective function values ​​are equal.

Assume that the optimal solution to the original problem is $x^{\ast }$ , dual variable$w^T=c_B^T{B}^{-1}$ .${}^{\_}$Because$x^{\ast }$ is the optimal solution, so
$$R=A^{T}w-c\le 0so$$ w \pmb
w$w$ is a feasible solution to the dual problem. At the same time$x$ is a basic feasible solution, which can be deduced as follows
$$c^{T}x=c_B^T{x}_B=c_B^T{B}^{-1}b=w^{T}b=b^{T}w$$
combined with the previous inference can be seen,$w$ is the optimal solution to the dual problem.

Dual problem application

There are many applications of dual problems, but most of them are used for theoretical derivation and proof. In practical applications, shadow prices are more common.

shadow price

According to the strong duality theorem, when the original problem and the dual problem have optimal solutions at the same time, the objective function values ​​​​of the two are equal, that is,
$$f=c^T{x}^{\ast }=z=b^T{w}^{\ast }$$
Written in the form of components
$$f=w_1^{\ast }{b}_1+w_2^{\ast }{b}_2+\cdot \cdot \cdot +w_m^{\ast }{b}_m$$
对$b_i$求导
$$\frac{\partial f}{\partial b_i}=w_i^{\ast },i=1,2,\cdot \cdot \cdot ,The\; right\; side\; of\; m$$
is obviously the optimal solution we get when solving the dual problem; while the left side can be intuitively understood as the gradient value of the objective function value relative to the resource variable. The smaller the value, the smaller the value of the objective function value that can be brought about by the increase of the corresponding resource. The greater the reduction.

Next, we will give an example to visually illustrate the meaning of shadow price.

原问题：
$$\begin{gathered} minf(x)=-5x{\_}_1-4x{\_}_2 \\ \text{s.t}{x}_1+3x{\_}_2+x_3=90 \\ 2x{\_}_1+x_2+x_4=80 \\ {x}_1+x_2+x_5=45 \\ {x}_1,x_2,x_3,x_4,x_5\ge 0Call \end{gathered}$$
ortools to solve

from ortools.linear_solver import pywraplp


if __name__ == '__main__':
    # 声明ortools求解器，使用SCIP算法
    solver = pywraplp.Solver.CreateSolver('SCIP')

    # 优化变量
    x1 = solver.NumVar(0, solver.infinity(), 'x1')
    x2 = solver.NumVar(0, solver.infinity(), 'x2')
    x3 = solver.NumVar(0, solver.infinity(), 'x3')
    x4 = solver.NumVar(0, solver.infinity(), 'x4')
    x5 = solver.NumVar(0, solver.infinity(), 'x5')

    # 目标函数
    solver.Minimize(-5 * x1 - 4 * x2)
    # 约束条件
    solver.Add(1 * x1 + 3 * x2 + x3 == 90)
    solver.Add(2 * x1 + 1 * x2 + x4 == 80)
    solver.Add(1 * x1 + 1 * x2 + x5 == 45)

    # 模型求解
    status = solver.Solve()

    # 模型求解成功, 打印结果
    if status == pywraplp.Solver.OPTIMAL:
        # 变量最优解
        print('x1: {}, x2: {}, x3: {}, x4: {}, x5: {}'.format(
            x1.solution_value(), x2.solution_value(), x3.solution_value(),
            x4.solution_value(), x5.solution_value()))

        # 最优目标函数值
        print('best_f =', solver.Objective().Value())

    else:
        print('not converge.')

After running the code, the optimal solution is

x1: 35.0, x2: 10.000000000000007, x3: 24.99999999999998, x4: -7.105427357601002e-15, x5: -7.105427357601002e-15
best_f = -215.00000000000003

对偶问题：
$$\begin{gathered} maxz(w)=90w{\_}_1+80w_2+45{in}_3 \\ \text{s.t}{w}_1+2w{\_}_2+w_3\le -5 \\ 3w_1+w_2+w_{e}3\le -4 \\ {w}_1,w_2,w_3\le 0 \end{gathered}$$
or call ortools to solve

from ortools.linear_solver import pywraplp


if __name__ == '__main__':
    # 声明ortools求解器，使用SCIP算法
    solver = pywraplp.Solver.CreateSolver('SCIP')

    # 优化变量
    x1 = solver.NumVar(-solver.infinity(), solver.infinity(), 'x1')
    x2 = solver.NumVar(-solver.infinity(), solver.infinity(), 'x2')
    x3 = solver.NumVar(-solver.infinity(), solver.infinity(), 'x3')

    # 目标函数
    solver.Maximize(90 * x1 + 80 * x2 + 45 * x3)
    # 约束条件
    solver.Add(1 * x1 + 2 * x2 + x3 <= -5)
    solver.Add(3 * x1 + 1 * x2 + x3 <= -4)
    solver.Add(1 * x1 <= 0)
    solver.Add(1 * x2 <= 0)
    solver.Add(1 * x3 <= 0)

    # 模型求解
    status = solver.Solve()

    # 模型求解成功, 打印结果
    if status == pywraplp.Solver.OPTIMAL:
        # 变量最优解
        print('x1: {}, x2: {}, x3: {} '.format(x1.solution_value(), x2.solution_value(), x3.solution_value()))

        # 最优目标函数值
        print('best_f =', solver.Objective().Value())

    else:
        print('not converge.')

After running the code, the optimal solution is

x1: 0.0, x2: -1.0000000000000002, x3: -3.0 
best_f = -215.00000000000006

Obviously, the optimal objective function values ​​are the same.

If the first constraint in the original problem code is modified: 90->91, that is, the first resource variable is increased by 1 unit, and the optimal solution is recalculated as

x1: 35.0, x2: 10.000000000000005, x3: 25.999999999999986, x4: -5.329070518200751e-15, x5: -5.329070518200751e-15
best_f = -215.0

At this time, the optimal objective function value has not changed; from the optimal solution of the dual problem, $w_1^{\ast }=0$ , which is logical.

If the second constraint in the original problem code is modified: 80->81, the optimal solution is recalculated as

x1: 36.0, x2: 9.000000000000004, x3: 26.99999999999999, x4: -3.552713678800501e-15, x5: -3.552713678800501e-15
best_f = -216.00000000000003

At this time, the optimal objective function value is reduced by 1 unit; from the optimal solution of the dual problem, $w_2^{\ast }=-1$ , is also logical.

If the third constraint in the original problem code is modified: 45->46, the optimal solution is recalculated as

x1: 34.00000000000001, x2: 11.999999999999995, x3: 20.000000000000007, x4: -8.881784197001252e-15, x5: -1.7763568394002505e-15
best_f = -218.00000000000003

At this time, the optimal objective function value is reduced by 3 units; from the optimal solution of the dual problem, $w_3^{\ast }=-3$ , so still no problem.

Recently, I happened to come across a case of trying to use shadow prices in an industrial scenario: Inventory Based Recommendation Algorithms
. Let me briefly talk about my understanding.

Let’s first describe the scene: Hema’s shopping cart interface can recommend some perishable items. If these items are not sold out on the same day, they will be scrapped or discounted the next day. The stock quantity of each item is limited. In the most ideal situation, all products should be sold out on the same day. So when each user enters the shopping cart page, which products should the system recommend?

The easiest strategy to think of is: calculate the value that each item can bring when it is recommended
$$ct{r}_i\ast cv{r}_i\ast r_i$$
Among them, $ct{r}_i$、$cv{r}_i$和$r_i$Respectively $Click-through\; rate,\; conversion\; rate\; and\; price\; of\; i$ item.

Once you have this value, you can arrange the products from large to small according to this value and recommend the top K.

A potential problem with this strategy is that the inventory of the product is not taken into account, which may cause some high-value products to be sold out early, while medium-value products are not sold out due to reasons such as too few recommendations.

In order to solve this problem, shadow prices can be introduced to fine-tune the above strategy:
$$ct{r}_i\ast cv{r}_i\ast (r_i-a_i)$$
where,$a_i$That is the iiThe shadow price corresponding to the inventory constraint of item$i\; .$

After adding the shadow price parameter, when the inventory of high-value products is sufficient, the shadow price of the product is 0, which does not affect the original recommendation strategy; when the inventory of high-value products becomes less, the product becomes scarce, and the shadow price will be greater than 0. , thereby lowering the ranking of high-value items to give way to other items of slightly lower value that are more in stock.

Theoretical application

Some of the more theoretical applications are briefly mentioned here:

(1) First, it provides a new method for solving linear programming problems: the dual simplex method;

(2) Secondly, it provides a new idea to prove that the original problem has no solution: Farkas’ lemma;

(3) The last one has some extensions: regardless of whether the original problem is a convex problem, its Lagrangian dual problem must be a convex optimization problem.

references

There is no such chapter in previous articles, but in the process of actual study and summary, I have actually referred to many articles or videos of great experts, and it is difficult to add a quote or link in a sentence to express respect for the original author. A tribute to the achievements, because many new cognitions are scattered in every corner of the article. Therefore, for convenience, they will be placed at this location in the future.

Dual problem example: https://weread.qq.com/web/reader/da232bb072012022da29f24kc7432af0210c74d97b01b1c

Why is the dual problem of linear programming needed: https://www.zhihu.com/question/26658861

Derivation of the dual problem: https://zhuanlan.zhihu.com/p/522590887

Lagrange duality: https://blog.csdn.net/frostime/article/details/90291392?depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromBaidu-1&utm_source=distribute.pc_relevant.none-task- blog-BlogCommendFromBaidu-1

Application of shadow price in Hema: https://ieeexplore.ieee.org/abstract/document/9378261