Matrix determinant calculation
Second-order determinant calculation
\[ det \left( \begin{matrix} a & b \\ c & d \end{matrix} \right ) = ad - bc \]
Third-order determinant calculation
\ [Det \ left (\ begin {matrix} a_ {11} & a_ {12} & a_ {13} \\ a_ {21} & a_ {22} & a_ {23} \\ a_ {31} & a_ { 32} & a_ {33} \ end {matrix} \ right) = a_ {11} a_ {22} a_ {33} + a_ {12} a_ {23} a_ {31} + a_ {13} a_ {21} a_ {32} - a_ {13
} a_ {22} a_ {31} -a_ {12} a_ {21} a_ {33} -a_ {11} a_ {23} a_ {32} \] formally expressed as: (red addend, subtrahend blue line)
Inverse matrix calculation
1. The method of undetermined coefficients
Order matrix \ (A = \ left (\ Matrix the begin {-1}. 1 & 2 & -3 \\ \ Matrix End {} \ right) \) , so that the solutions required for the inverse matrix \ (A ^ {- 1 =} \ left (\} the begin {Matrix A & B & C D \\ \ Matrix End {} \ right) \) . There \ (A * A ^ {-}. 1 the I = \) , obtained equations
\ [\ begin {matrix} a + 2c = 1 \\ b + 2d = 0 \\ -a - 3c = 0 \\ -b - 3d = 1 \ end { matrix} \]
Thereby Solutions \ (A =. 3, B = 2, C = -1, D -1 = \) , i.e.
\ [A ^ {- 1} = \ left (\ begin {matrix} 3 & 2 \\ -1 -1 & \ matrix End {} \ right) \]
2. elementary transformation inverse matrix
Likewise, let the matrix \ (A = \ left (\ Matrix the begin {-1}. 1 & 2 & -3 \\ \ Matrix End {} \ right) \) , write it augmented matrix \ (A | E \ ) , i.e. the matrix \ (a \) on the right side is placed a unit matrix of the same order, to give a new matrix.
\ [A | E = \ left
(\ begin {matrix} 1 & 2 & 1 & 0 \\ -1 & -3 & 0 & 1 \ end {matrix} \ right) \] subjected to elementary transformation, so that the original \ (a \) matrix transformation be at the matrix, the matrix of the original is converted become \ (a ^ {-}. 1 \) , the transformation matrix is:
\ [\ left (\}. 1 the begin {matrix & 0 & 3 & 2 \\ 0 & 1 & -1 & -1 \ end {matrix} \ right) \]
Common gradient formula
\[ (1)\ f(x) = C(常数), \nabla f(x) = 0, 即\nabla C = 0 \]
\[ (2)\ f(x) = b^{T}x, \nabla f(x) = b \]
\[ (3)\ f(x) = x^{T}x, \nabla f(x) = 2x \]
\[ (4)\ f(x) = x^{T}Qx (Q^T=Q), \nabla f(x) = 2Qx \]
Taylor expansion formula
Set \ (f: R ^ n \ rightarrow R \) second order can be turned, in \ (x ^ * \) within the field of
first-order Taylor expansion as
\ [f (x) = f (x ^ *) + ( \ nabla f (x ^ *) ) ^ T (xx ^ *) + o (|| xx ^ * ||) \]
Expanding to second order Taylor
\ [f (x) = f (x ^ *) + (\ nabla f (x ^ *)) ^ T (xx ^ *) + \ frac {1} {2} (xx ^ * ) ^ T \ nabla ^ 2f ( x ^ *) (xx ^ *) + o (|| xx ^ * ||) \]
Convex sets
If the set \ (D \) any two points in the wire segments belong \ (D \) , called \ (D \) is convex.
Two \ (x ^ 1, x ^ 2 \) wire segments at any point can be expressed as \ (x = ax ^ 1 + (1-a) x ^ 2, a \ in [0,1] \)
Set \ (A, B \ subseteq R ^ n \) is a convex set, then \ (A \ bigcap B, A + B, A - B \) is convex. Wherein \ (A + B: = {A + B: A \ in A, B \ in B} \) , \ (AB: = {ab &: A \ in A, B \ in B} \) , attention \ ( A \ bigcup B \) is not necessarily convex.
\ (D \) is a convex set \ (\ Longleftrightarrow D \) convex combination of any of the plurality of finite points belong \ (D \) .
Convex function
is defined: A collecting \ (D \ subseteq R ^ n \) is a convex set, the function \ (F: D \ rightarrow R & lt \) , if the \ (\ forall x ^ 1, x ^ 2 \ in D, \ alpha \ in (0,1) \) , are:
\ [F (\ X ^. 1 Alpha + (. 1 - \ Alpha) X ^ 2) \ Leq \ Alpha F (X ^. 1) + (l- \ Alpha) f (x ^ 2) \]
called \ (F \) is convex set \ (D \) convex function on.
Suppose \ (f (x), g (x) \) are convex functions, proof \ (max \ lbrace f (x ), g (x) \ rbrace \) is a convex function.
证明：
\(f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2), g(tx_1 + (1-t)x_2) \leq tg(x_1) + (1-t)g(x_2)\)
\(max \lbrace f(tx_1+(1-t)x_2), g(tx_1, (1-t)x_2) \rbrace \leq max\lbrace tf(x_1) + (1-t)f(x_2), tg(x_1) + (1-t)g(x_2) \rbrace \leq t max\lbrace f(x_1), g(x_1) \rbrace + (1-t) max\lbrace f(x_2), g(x_2) \rbrace\)
Convex function is determined Theorem
disposed \ (D \ subseteq R ^ n \) is non-empty open convex set, the function \ (f: D \ rightarrow R \) in \ (D \) twice differentiable on the
(I) \ (F \) in \ (D \) on the convex function \ (\ Longleftrightarrow \ forall x \ in D, \ nabla ^ 2f (x) \) Semidefinite
(ii) if \ (\ forall x \ in D , \ nabla ^ 2 f (x) \) positive definite, the \ (F \) in (D \) \ a strictly convex function
For Convex Programming
$
min F (X)
$
$
ST G_i (X) \ Leq 0, I = 1,2, L, m
$
If \ (\ overline {X} \ in D, F \ ^. 1 in C \) , then \ (\ overline {x} \ ) is necessary and sufficient conditions for the optimal solution for the formula \ (\ forall x \ in D \) , there are
\ [(x- \ overline {x }) ^ T \ nabla f (\ overline {x}) \ ge 0 \]
Precise one-dimensional search
Success or failure Act
Step 1: Select the initial point \ (X \ in R & lt \) , the initial step \ (h \ gt 0 \) and precision \ (\ Epsilon \ gt 0 \) , \ (\ varphi_1 = F (X) \)
Step 2: calculation \ (\ varphi_2 = f (x
+ h) \) step 3: If \ (\ varphi_2 <\ varphi_1 \) , the search is successful, go to step 4; otherwise, the search fails, go to step 5.
Step 4: Order \ (X: = X + H \) , \ (\ varphi_1: = \ varphi_2, H: 2H = \) , go to step 2.
Step 5: Determine \ (| H | <\ Epsilon \) ? If \ (| h | <\ epsilon \) to stop the iteration, \ (X * = X ^ \) ; else let \ (h = \ frac {-h } {4} \) go to step 2.
0.618
Using four points sequentially shortened interval, when the interval length is sufficiently small, may be taken as the approximate midpoint of the interval points minimum point. The four points are selected \ (A, B, x_1, x_2 \) , wherein, \ (A, B \) , respectively lower and upper bounds of the interval. \ (x_1, x_2 \) is calculated in line with 0.618 criteria, as follows:
\ [\} the begin Matrix x_1 = {A + 0.382 (B-A) + A \\ x_2 = 0.618 (B-A) \} End {Matrix \ ]
dichotomy
Since we assume that the given function is a convex function section, so that there must be such that it is within the interval range derivative is 0, the point is the minimum point. It can be calculated each time interval midpoint derivative value, in order to narrow the range in which the interval is sufficiently small, so that the midpoint of the interval is local minimum.
Step 1: Calculate \ (x_0 = \ frac {a
+ b} {2} \) Step 2: If \ (F ^ { '} (x_0) \ lt 0 \) , so \ (A = x_0 \) , transfer step 3;
if \ (F ^ { '} (x_0) \ gt 0 \) , so \ (B = x_0 \) , go to step 3;
if \ (F ^ {'} (x_0) = 0 \) , stop , \ (X ^ * = x_0 \) ;
step 3: If \ (| BA | \ lt \ Epsilon \) , then \ (X ^ * = \ FRAC {A + B} {2} \) , stop, or , go to step 1.
Newton's method
Step 1: Given initial point \ (x_1, \ Epsilon \ gt 0 \) , so \ (k = 1 \)
Step 2: Calculate \ (x_ {k + 1} = x_ {k} - \ frac {f ^ { '} (x_k)} {
f ^ {' '} (x_k)} \) step 3: If \ (| F ^ {'} (X_ {K +. 1}) | \ lt \ Epsilon \) , is stopped, \ (x ^ * \ approx x_ {k + 1} \)
Quasi Newton method
needs to construct \ (H_k \) , \ (H_k \) several principles need to be met are:
Quasi-Newton property
$ \ X_k of Delta + = H_ {K}. 1 \ nabla G_k \ (, where \) H_ {K} $ +. 1 is not unique, the performance of a quasi-Newton type nature, condition or equation called.
Quadratic convergence
\ (f (x) = \ frac {1} {2} x ^ TAx + b ^ Tx + c, A symmetric positive definite \)
the algorithm \ (\ n-) epoch positive quadratic function, up \ (n- \) time to reach minimum point.stability
In the algorithm iterates \ (x ^ k \) at a selectable step size, so that the next iteration points \ (x ^ {k + 1 } of the function value decreases, the algorithm is called stable. \)
Interpolation
Using the method for finding success or failure of "high, low, high," three, respectively, as a point within the interval and the interval, then the three point configuration two (three, four) times equation, the extreme point as the fourth method points, and to further shorten the interval by the function of these four points.
Step 1] Method by the success or failure) to find the "high - low - high" three: i.e. three satisfy
\ [x_1 \ lt x_2 \ lt x_3, \ f (x_1) \ gt f (x_2) \ lt f (x_3 ) \]
step 2: determining quadratic interpolation polynomial \ (P (X) = X + A_1 A_2 a_0 + X ^ 2 \) , obtains \ (P \) of the minimum point \ (\ overline {x} \ ) (by \ (P (x_1) \ gt P (x_2) \ lt P (X_3) \) ), so \ (A_2 \ gt 0 \) , \ (\ overline {X} \ in [x_1, X_3] \)
step 3: If \ (| x_2 - \ overline {X} | \ lt \ Epsilon \) , then the iteration ends, take \ (X ^ * = \ overline {X} \) , or at a point \ (x_1, x_2, x_3, \ overline {x} \ ) , select enable (F \) \ minimum point as a new \ (x_2 \) , and the new \ (x_1, x_3 \) are each a new \ (x_2 \) two right and left vicinity, go to step 2, the iteration continues until a termination condition is met.
Steepest descent method
Step 1: Select the initial point \ (X, \ Epsilon \ gt 0 \) , and let \ (K = 1 \) .
Step 2: If \ (|| \ nabla F (X ^ K) || \ Leq \ Epsilon \) , approximate stagnation point \ (X ^ K \) , otherwise, go to Step 3.
Step 3: Order \ (d ^ k = - \ nabla f (x ^
k) \) step 4: one-dimensional search is determined by the exact optimum step size \ (\ lambda_k = Arg \ min \ F (K + X ^ \ ^ K the lambda D) \) . Order \ (^ {X}. 1 + K = K + X ^ \ ^ _kd the lambda K, K: = K +. 1 \) , go to step 2
Newton's method
Requests the function \ (F \) minimum point for a given error limit \ (\ Epsilon \) .
Step 1: Select the initial point \ (X ^ 1 \) , calculated \ (f_1 = f (x ^ 1), k = 1 \)
step 2: If \ (|| \ nabla F (X ^ K) || \ Leq \ Epsilon \) , is stopped, to obtain approximately stationary points \ (X ^ K \) , otherwise, go to step 3.
step 3 : calculated search direction \ (d ^ k = - (
\ nabla ^ 2f (x ^ k)) ^ {- 1} \ nabla f (x ^ k) \) step 4: order \ (x ^ {k + 1 } X + K ^ D = ^ K, K = K +. 1 \) , go to step 2
Damped Newton method
will be the method of Newton iteration Alternatively, addition of a precise dimensional search \ (min F (K + X ^ \ ^ K the lambda D) \) , to obtain the optimum step length \ (\ lambda_k \) , to give the next iteration point \ (x ^ {k + 1 } = x ^ k + \ lambda_k d ^ k \)
Newton method using the damper element of n - definite minimum points a quadratic function, starting from any initial point, iteration can achieve minimum point.
FR Conjugate Gradient Method
Step 1: Select the initial point \ (x ^ 1 \)
Step 2: If \ (|| || G_1 \ Leq \ Epsilon \) , is stopped, to obtain approximately stationary points \ (x ^ 1 \) , otherwise, go to Step 3
step 3: \ (p ^ 1 = -g_1,
k = 1 \) step 4: exact line search to find the optimal step size \ (\ lambda_k \) , so \ (x ^ {k + 1 } = x ^ k + \ lambda_kp ^ k \)
step 5: If \ (|| + G_ {K}. 1 || \ Leq \ Epsilon \) , is stopped, to obtain approximately stationary points \ (K +. 1 {X} ^ \) , or go to step 6
step 6: If \ (n-K = \) , so \ (= X ^ X ^. 1. 1 + {K}, P = ^. 1. 1 + K -g_ {}, K =. 1 \) , go to step 4; otherwise, proceed to step 7
step 7: calculation \ (a_k = \ frac {|| g_ {k + 1} || ^ 2} {|| g_k || ^ 2}, p ^ {k + 1} = - g_ K +. 1} + {^ a_kp K, K = K +. 1 \) , go to step 4
Variable metric method --DFP algorithm
Step 1: Select the initial point \ (X ^ 1 \) , initial matrix \ (H_1 I_n = \) , \ (\ Epsilon \ gt 0 \)
Step 2: If \ (|| g_1 || \ leq \ epsilon \ ) , is stopped, to obtain approximately stationary points \ (X ^. 1 \) , otherwise, go to step 3.
step 3: \ (p ^ 1 = -H_1g_1,
k = 1 \) step 4: exact line search to find the optimal step size \ (\ _K the lambda \) , so \ (x ^ {k + 1
} = x ^ k + \ lambda_k p ^ k \) step 5: If \ (|| g_ {k + 1 } || \ leq \ epsilon \) , is stopped, to obtain approximately stationary points \ (K +. 1 {X} ^ \) , otherwise turn to step 6
step 6: If \ (n-K = \) , so \ (x ^ 1 = x ^ {k + 1 }, P = ^. 1. 1 + K -g_ {}, K =. 1 \) , go to step 4, otherwise go to step 7
step 7: order \ (\ Delta x_k = x ^ {k + 1} - x ^ k, \ = G_ G_k of Delta + {K}. 1 - G_k, RJc = H_k \ G_k of Delta \) , is calculated:
\ [H_ {k + 1} = H_k + \ frac {\ Delta x_k (\ Delta x_k) ^ T} {(\ Delta x_k) ^ T \ Delta g_k} - \ frac {r_k (r_k) ^ T} {( \ Delta g_k) ^ T H_k \
Delta g_k} \] and \ (P ^ {K +. 1} = - H_ {K +. 1} G_ {K +. 1} \) , so \ (k = k + 1 \ ) , proceed to step 4
Two-stage process first requires the introduction of the original variable in the equation artificial variables and artificial variables are greater than or equal to 0, then the simplex method for solving such artificial Variables and minimum conditions, then the modified simplex tableau, artificial variable omitted, continue optimal solution using simplex method to obtain the objective function. It is noted that, when selecting the variables into groups, the number of test select, i.e., \ (\ sigma_j \) is greater than 0, and a maximum. When a variable group selected from, select \ (\ Theta \) is greater than 0 and a minimum. When the values are equal, the index selected from the group variable maximum, in variable selection table minimal.
Large \ (M \) Act
Introducing artificial constraints variables \ (X_a \) , while modifying the objective function, plus penalty terms in the original target \ (^ Me TX A \) , where \ (M \) is a sufficiently large positive number.
Dual programming
Dual simplex method
The basic idea of the dual simplex method: A basic solution even feasible from (P), and in ensuring the dual feasible conditions, and gradually make the original problem fundamental solution can not
row disappears (ie x non-negative), until the original a basic feasible solution of the problem so far, and this basic feasible solution is the optimal solution to the original problem.
对偶单纯形方法与原单纯形方法主要的区别就是，先计算\(\overline{b}\)，找出其中小于0，且最小的一个作为离基变量，然后用\(\sigma_j\)除以对应的行，得到参考值，选择参考值中大于0，且最小的一个作为进基变量。目标是使\(\overline{b}\)值均大于等于0.
K-T点计算，一阶最优性条件
\[ \begin{matrix} \nabla f(\overline{x}) - \sum_{i \in I(\overline{x})}w_i\nabla g_i(\overline{x}) - \sum_{j=1}^{l} v_j \nabla h_j(\overline{x}) = 0 \\ w_i \geq 0,i \in I(\overline{x}) \end{matrix} \]
\[ \begin{matrix} \nabla f(\overline{x}) - \sum_{i=1}^{m}w_i\nabla g_i(\overline{x})-\sum_{j=1}^l v_j \nabla h_j(\overline{x}) = 0 \\ w_i \geq 0, i=1,...,m \\ w_i g_i(\overline{x}) = 0,i=1,...,m \end{matrix} \]
满足以上两个条件中的任意一个即为K-T点。若要求K-T点，用下式，若要验证，用上式。其中，\(g_i\)为大于约束，\(h_i\)为等式约束。
外点罚函数法
构造罚函数：\(min F(x,M_k)=f(x)+M_k p(x)\)。
其中\(M_k\)逐渐趋于无穷
优化函数为
\[ \begin{matrix} min\ f(x) \\ s.t.\ g_i(x) \geq 0, i=1,...,m \\ h_j(x) = 0,j=1,...,m \end{matrix} \]
则\(p(x) = \sum_{i=1}^{m}(min\lbrace g_i(x),0 \rbrace )^2+\sum_{j=1}^l h_j^2(x)\)，可用\(x\)来表示\(M\)，然后让\(M\)趋于无穷，以此求得\(x\)。
外点罚函数法性质
- \(F(x^{k+1}, M_{k+1}) \geq F(x^k, M_k)\)
- \(p(x^{k+1}) \leq p(x^k)\)
- \(f(x^{k+1}) \geq f(x^k)\)
外点罚函数法不等式证明
\(F(x^{k+1}, M_{k+1}) \geq F(x^k, M_k)\)
对于由外点法产生的点列\(\lbrace x^k \rbrace\)，\(k \geq 1\)，总有：
$
F(x^{k+1}, M_{k+1}) = f(x^{k+1}) + M_{k+1}p(x^{k+1}) \geq f(x^{k+1}) + M_kp(x^{k+1})
$
=$
F(x^{k+1}, M_k)
$
$
\geq F(x^k,M_k)
$\(p(x^{k+1}) \leq p(x^k)\)
$
f(x^{k+1}) + M_kp(x^{k+1}) = F(x^{k+1}, M_k) \geq F(x^k, M_k) = f(x^k)+M_kp(x^k)
$
$
f(x^{k+1}) - f(x^k) \geq M_k[p(x^k)-p(x^{k+1})]
$
$
f(x^k)+M_{k+1}p(x^k)=F(x^k, M_{k+1})\geq F(x^{k+1}, M_{k+1})=f(x^{k+1})+M_{k+1}p(x^{k+1})
$
$
f(x^k)-f(x^{k+1}) \geq M_{k+1}[p(x^{k+1})-p(x^k)]
$
$
0 \geq (M_{k+1}-M_k)[p(x^{k+1})-p(x^k)] 已知M_{k+1}\geq M_k
$
$
p(x^{k+1}) \leq p(x^k)
$\(f(x^{k+1}) \geq f(x^k)\)
\(f(x^{k+1}) + M_kp(x^{k+1})=F(x^{k+1},M_k)\geq F(x^k,M_k)=f(x^k)+M_kp(x^k)\)
\(已知p(x^{k+1})\leq p(x^k)\)
\(f(x^{k+1})\geq f(x^k)\)
内点罚函数法
构造罚函数：\(F(x,r)=f(x)+rB(x)\),只可用于不等书约束的条件，要求大于等于0
\[ \begin{matrix} B(x) = \sum_{i=1}^m \frac{1}{g_i(x)} \\ B(x) = -\sum_{i=1}^m ln(g_i(x)) \end{matrix} \]
内点罚函数法性质
- \(F(x^{k+1}, r_{k+1}) \leq F(x^k, r_k)\)
- \(B(x^{k+1}) \geq B(x^k)\)
- \(f(x^{k+1}) \leq f(x^k)\)
约坦狄克可行方向法
步骤1：给定初始可行点\(x^0\)，令\(k=0\)
步骤2：在点\(x^k\)处将\(A\)和\(b\)分解成
\[ A=\left( \begin{matrix} A_1 \\ A_2 \end{matrix} \right) \]
\[ b=\left( \begin{matrix} b_1 \\ b_2 \end{matrix} \right) \]
使得\(A_1x^k=b, A_2x^k\gt b_2\)，计算\(\nabla f(x^k)\)。
步骤3：求解线性规划$
\begin{matrix}
min(\nabla f(x^k))^Td \
s.t. A_1d \geq 0 \
Cd = 0\
|d_j| \leq 1, \forall j
\end{matrix}
\(得到最优解\)d^k$
步骤4：若\((\nabla f(x^k))^Td^k = 0\)，则算法结束，\(x^k\)是K-T点，否则转步骤5
步骤5：利用(*)式计算\(\lambda_{max}\)，求解一维搜索问题
\[ \begin{matrix} min\ f(x^k+\lambda d^k) \\ s.t.\ 0\leq \lambda \leq \lambda_{max} \end{matrix} \]
得到极小点\(\lambda _k\)，令\(x^{k+1}=x^k+\lambda _kd^k,k=k+1\)，转步骤2.
$
\lambda _{max}=\left \lbrace
\begin{matrix}
min\lbrace \frac{\overline{b_i}}{\overline{d_i}}| \overline{d_i} \lt 0\rbrace if \exist\overline{d_i} \lt 0 \
+\infty if \overline{d} \geq 0
\end{matrix}
\right.
$ (*)
$
\overline d = A_2d^k , \overline b = b_2 - A_2x^k
$
注：单纯形法，\(\sigma_j\)选择大于0且值最大的进基，\(\theta\)选择大于0且值s最小的进基。
对偶单纯形，先找离基再找进基。\(\overline{b}\)选择小于0且最小的一个离基，\(\sigma\)除以对应的行，所得大于0且最小的进基。