Matrix guide (II)

Paper undertake Part  HTTPS: // zhuanlan.zhihu.com/p/24 709 748 , in terms of matrix matrix derivation surgery. Lowercase x represents a scalar, bold lowercase letters represent column vectors X represents uppercase matrix. Matrix derivation using the quantization matrix idea, a method often used in second order optimization problem is solved.

First, let's look at the definition of pondering. Matrix derivative of the matrix, the definition of what kind of needs? First, the matrix F (p × q) derivative of the matrix X (m × n) should contain all mnpq a partial derivatives , so that no loss of information; second, derivative and derivative have brief contact, as calculated derivative and Application this contact is required; third, derivatives concise algorithm starting from the whole. We define a vector (p × 1) of the vector (m × 1) a derivative of (m × p), there ; redefinition matrix (column-priority) the quantization (mn × 1), and define the matrix F matrix X derivative (mn × pq). Derivative and differential linked . Some explanations are as follows:

1. According to this definition, the amount of labeled derivative of f matrix X (m × n) is a mn × 1 vector, is not compatible with the definition of the article, but both easily interchangeable. To avoid confusion, a symbol represents Part defined m × n matrix, there . Although the technique may be used Benpian scalar matrix derivation this particular case, but the use of the article is more convenient in the art. Readers equivalent transformation by the two methods may test articles on the study.
2. Scalar second derivative matrix, also called the Hessian matrix, defined as (mn × mn), it is a symmetric matrix. Vectors or matrix derivation Hessian matrix can be obtained, but the matrix starting easier.
3. When the derivative matrix is vectorized, the drawbacks is that the destruction of the structure of the matrix to some extent, will lead to the results become complicated form; benefits of multivariate calculus regarding gradient, Hessian matrix conclusions can still be used up, simply matrix vectorization. For example optimization problem, update Newton's method to meet .
4. In profile, the matrix derivatives there are other definitions of matrices, such as (mp × nq), or (mp × nq), it is compatible with scalar Part of the definition of a derivative matrix, but contact differential and derivative ( dF is equal to the m × n sub-block one by one, respectively the inner product dX) concise enough, not convenient calculation and application. [5] as defined above are reviewed, and the definition of criticism that they are bad, can cooperate with the differential operation is well defined.

 

Then to establish algorithms. To contact still use derivative and differential , differentiated approach and the same article, to give the derivative from the differential needs some vectorization techniques:

1. Linear: .
2. Matrix multiplication: where represents the Kronecker product, A (m × n) and B (p × q) is the Kronecker product (mp × nq). This formula proves See "Matrix Analysis and Applications" on page 107-108 of Zhang Xian.
3. Transpose: , A is an m × n matrix, wherein (mn × mn) is a switching matrix (commutation matrix), the column will be prioritized to the priority quantization row becomes the quantization. For example .
4. Element-wise multiplication: wherein (mn × mn) is the element A (column-by) aligned diagonal matrix.

 

Look can be asserted if the matrix function F is a matrix X by subtraction multiplication, inverse, determinant, element-wise arithmetic functions such as configuration, using the appropriate algorithm for F differentiating do to quantify and tips to other items exchanged into vec (dX) left, derivative and differential control of contact , i.e., the derivative can be obtained.

特别地，若矩阵退化为向量，对照导数与微分的联系，即能得到导数。

 

再谈一谈复合：假设已求得，而Y是X的函数，如何求呢？从导数与微分的联系入手，，可以推出链式法则。

和标量对矩阵的导数相比，矩阵对矩阵的导数形式更加复杂，从不同角度出发常会得到形式不同的结果。有一些Kronecker积和交换矩阵相关的恒等式，可用来做等价变形：

1. 。
2. 。
3. 。可以对求导来证明，一方面，直接求导得到；另一方面，引入，有，用链式法则得到。
4. 。
5. ，A是m×n矩阵，B是p×q矩阵。可以对做向量化来证明，一方面，；另一方面，。

 

接下来演示一些算例。

例1：，X是m×n矩阵，求。

解：先求微分：，再做向量化，使用矩阵乘法的技巧，注意在dX右侧添加单位阵：，对照导数与微分的联系得到。

特例：如果X退化为向量，即，则根据向量的导数与微分的关系，得到。

 

例2：，X是n×n矩阵，求和。

解：使用上篇中的技术可求得。为求，先求微分：，再做向量化，使用转置和矩阵乘法的技巧，对照导数与微分的联系，得到，注意它是对称矩阵。在是对称矩阵时，可简化为。

 

例3：，A是l×m矩阵，X是m×n矩阵，B是n×p矩阵，exp为逐元素函数，求。

解：先求微分：，再做向量化，使用矩阵乘法的技巧：，再用逐元素乘法的技巧：，再用矩阵乘法的技巧：，对照导数与微分的联系得到。

 

例4【一元logistic回归】：，求和。其中是取值0或1的标量，是列向量。

解：使用上篇中的技术可求得，其中 为sigmoid函数。为求，先求微分：，其中为sigmoid函数的导数，对照导数与微分的联系，得到。

推广：样本，，求和。有两种方法，方法一：先对每个样本求导，然后相加；方法二：定义矩阵，向量，将写成矩阵形式，进而可以使用上篇中的技术求得。为求，先求微分，再用逐元素乘法的技巧：，对照导数与微分的联系，得到。

 

例5【多元logistic回归】：，求和。其中其中是除一个元素为1外其它元素为0的列向量，是矩阵，是列向量，是标量。

解：上篇中已求得。为求，先求微分：定义，，注意这里化简去掉逐元素乘法，第一项中，第二项中。定义矩阵，，做向量化并使用矩阵乘法的技巧，得到。

 

最后做个总结。我们发展了从整体出发的矩阵求导的技术，导数与微分的联系是计算的枢纽，标量对矩阵的导数与微分的联系是，先对f求微分，再使用迹技巧可求得导数，特别地，标量对向量的导数与微分的联系是；矩阵对矩阵的导数与微分的联系是，先对F求微分，再使用向量化的技巧可求得导数，特别地，向量对向量的导数与微分的联系是。

 

 

参考资料：

1. Zhang Xian up.  Matrix Analysis and Applications Tsinghua University Press Ltd, 2004.
2. Fackler, Paul L. "Notes on matrix calculus." North Carolina State University(2005).
3. Petersen, Kaare Brandt, and Michael Syskind Pedersen. "The matrix cookbook." Technical University of Denmark 7 (2008): 15.
4. HU, Pili. "Matrix Calculus: Derivation and Simple Application." (2012).
5. Magnus, Jan R., and Heinz Neudecker. "Matrix Differential Calculus with Applications in Statistics and Econometrics." Wiley, 2019.