本文主要对在机器学习中常用矩阵求导方法做以总结，以实用为主。同时感谢大佬@刘建平Pinard 的博文，浅显易懂，更加全面的资料请见底部参考文献 。
向量可以看做一组标量的排列集合 ，因此向量的求导可视为对向量中的标量逐个求导 ，这是向量求导的主要思路 。而矩阵的求导方式在此基础上继续衍生。
此处，用
x
x
x
x
\textbf{x}
x
X
X
X 。根据标量、向量和矩阵的组合，以下一共罗列了9种，其中在机器学习中常用的是①标量对向量 和②标量对矩阵 的求导。
自变量\因变量
标量
y
y
y
向量y
矩阵Y
标量
x
x
x
∂
y
∂
x
\frac{\partial y}{\partial x}
∂ x ∂ y
∂
y
∂
x
\frac{\partial \textbf{y}}{\partial x}
∂ x ∂ y
∂
Y
∂
x
\frac{\partial Y}{\partial x}
∂ x ∂ Y
向量x
∂
y
∂
x
\frac{\partial y}{\partial \textbf{x}}
∂ x ∂ y
∂
y
∂
x
\frac{\partial \textbf{y}}{\partial \textbf{x}}
∂ x ∂ y
∂
Y
∂
x
\frac{\partial Y}{\partial \textbf{x}}
∂ x ∂ Y
矩阵X
∂
y
∂
X
\frac{\partial y}{\partial X}
∂ X ∂ y
∂
y
∂
X
\frac{\partial \textbf{y}}{\partial X}
∂ X ∂ y
∂
Y
∂
X
\frac{\partial Y}{\partial X}
∂ X ∂ Y
求导的结果的维度是分子的维度 。这里，我自己的理解是将分子 的顺序排在最前。
【例如】有
m
×
1
m\times1
m × 1
y
\textbf{y}
y
1
×
1
1\times1
1 × 1
x
x
x
∂
y
∂
x
\frac{\partial \textbf{y}}{\partial x}
∂ x ∂ y ①分子布局 ：“优先”分子
y
\textbf{y}
y
y
\textbf{y}
y
x
x
x
(
m
×
1
)
×
(
1
×
1
)
(m\times1)\times (1\times1)
( m × 1 ) × ( 1 × 1 )
m
×
1
m\times1
m × 1 ②分母布局 ：“优先”分母
x
x
x
x
x
x
y
T
\textbf{y}^T
y T
(
1
×
1
)
×
(
1
×
m
)
(1\times 1)\times (1\times m)
( 1 × 1 ) × ( 1 × m )
1
×
m
1\times m
1 × m
求导的结果的维度是分母的维度 。
自变量\因变量
1
×
1
1\times1
1 × 1
y
y
y
m
×
1
m\times1
m × 1
y
\textbf{y}
y
p
×
q
p\times q
p × q
Y
Y
Y
1
×
1
1\times1
1 × 1
x
x
x ——
∂
y
∂
x
\frac{\partial \textbf{y}}{\partial x}
∂ x ∂ y 分子布局 ：得
m
×
1
m\times 1
m × 1 默认)
1
×
m
1\times m
1 × m
∂
Y
∂
x
\frac{\partial Y}{\partial x}
∂ x ∂ Y 分子布局 ：得
p
×
q
p\times q
p × q 默认)
q
×
p
q\times p
q × p
n
×
1
n\times1
n × 1
x
\textbf{x}
x
∂
y
∂
x
\frac{\partial y}{\partial \textbf{x}}
∂ x ∂ y
1
×
n
1\times n
1 × n 分母布局 ：得
n
×
1
n\times 1
n × 1 默认)
∂
y
∂
x
\frac{\partial \textbf{y}}{\partial \textbf{x}}
∂ x ∂ y 分子布局 ：得
m
×
n
m\times n
m × n 雅克比矩阵(默认)
n
×
m
n\times m
n × m 梯度矩阵——
n
×
m
n\times m
n × m
X
X
X
∂
y
∂
X
\frac{\partial y}{\partial X}
∂ X ∂ y
m
×
n
m\times n
m × n 分母布局 ：得
n
×
m
n\times m
n × m 默认)——
——
当对向量或矩阵求导时，对其分量进行逐个求导。
【例如】：对于实值函数
y
=
x
T
A
x
y=x^TAx
y = x T A x
x
∈
R
n
×
1
,
A
∈
R
n
×
n
x\in R^{n\times 1},A\in R^{n\times n}
x ∈ R n × 1 , A ∈ R n × n
∂
x
T
A
x
∂
x
\frac{\partial x^TAx}{\partial x}
∂ x ∂ x T A x
x
\textbf{x}
x
n
n
n
k
k
k
∂
x
T
A
x
∂
x
k
=
∂
∑
i
=
1
n
∑
j
=
1
n
x
i
A
i
j
x
j
∂
x
k
=
∑
i
=
1
n
∑
j
=
1
n
δ
i
k
A
i
j
x
j
+
∑
i
=
1
n
∑
j
=
1
n
x
i
A
i
j
δ
j
k
\frac{\partial x^TAx}{\partial x_k}=\frac{\partial \sum_{i=1}^n \sum_{j=1}^n x_iA_{ij}x_j}{\partial x_k}= \sum_{i=1}^n\sum_{j=1}^n \delta_{ik} A_{ij}x_j+ \sum_{i=1}^n\sum_{j=1}^n x_i A_{ij}\delta_{jk}
∂ x k ∂ x T A x = ∂ x k ∂ ∑ i = 1 n ∑ j = 1 n x i A i j x j = i = 1 ∑ n j = 1 ∑ n δ i k A i j x j + i = 1 ∑ n j = 1 ∑ n x i A i j δ j k
δ
i
k
=
1
，
i
f
i
=
k
\delta_{ik}=1，if\; i=k
δ i k = 1 ， i f i = k
δ
i
k
=
0
，
i
f
i
≠
k
\delta_{ik}=0，if\; i\not=k
δ i k = 0 ， i f i = k
k
=
1
,
2
,
.
.
.
,
n
k=1,2,...,n
k = 1 , 2 , . . . , n
∂
x
T
A
x
∂
x
k
=
∑
j
=
1
n
A
k
j
x
j
+
∑
i
=
1
n
x
i
A
i
k
=
A
x
+
x
T
A
=
A
x
+
A
T
x
\frac{\partial x^TAx}{\partial x_k}= \sum_{j=1}^n A_{kj}x_j+ \sum_{i=1}^n x_i A_{ik}=Ax+x^TA=Ax+A^Tx
∂ x k ∂ x T A x = j = 1 ∑ n A k j x j + i = 1 ∑ n x i A i k = A x + x T A = A x + A T x
对于实值函数
y
=
a
T
X
b
y=a^TXb
y = a T X b
a
∈
R
n
×
1
,
b
∈
R
m
×
1
,
X
∈
R
n
×
m
a\in R^{n\times 1},b\in R^{m\times 1},X\in R^{n\times m}
a ∈ R n × 1 , b ∈ R m × 1 , X ∈ R n × m
∂
a
T
X
b
∂
X
\frac{\partial a^TXb}{\partial X}
∂ X ∂ a T X b
X
X
X
n
×
m
n\times m
n × m
X
i
j
X_{ij}
X i j
∂
a
T
X
b
∂
X
=
∂
∑
p
=
1
n
∑
q
=
1
m
a
p
X
p
q
b
q
∂
X
i
j
=
δ
i
p
δ
j
q
a
i
b
j
=
a
i
b
j
\frac{\partial a^TXb}{\partial X}=\frac{\partial \sum_{p=1}^n \sum_{q=1}^m a_p X_{pq}b_q}{\partial X_{ij}}= \delta_{ip}\delta_{jq} a_i b_j=a_i b_j
∂ X ∂ a T X b = ∂ X i j ∂ ∑ p = 1 n ∑ q = 1 m a p X p q b q = δ i p δ j q a i b j = a i b j
i
=
1
,
2
,
.
.
.
,
n
且
j
=
1
,
2
,
.
.
,
m
i=1,2,...,n且j=1,2,..,m
i = 1 , 2 , . . . , n 且 j = 1 , 2 , . . , m
n
×
m
n\times m
n × m
∂
a
T
X
b
∂
X
=
a
b
T
\frac{\partial a^TXb}{\partial X}=ab^T
∂ X ∂ a T X b = a b T
需要逐个对向量和矩阵的每个元素进行求导，然后在排列起来。对于复杂的式子难以做到且效率不高，因此需要一个整体求导解法。
回顾一下高等数学中从极限引出的导数和微分。其中
Δ
y
\Delta y
Δ y
d
y
dy
d y
因此导数和微分的关系如下：
d
y
=
f
′
(
x
)
d
x
，
d
x
=
Δ
x
dy=f'(x) dx，dx=\Delta x
d y = f ′ ( x ) d x ， d x = Δ x
f
′
(
x
)
=
d
y
d
x
f'(x)=\frac{dy}{dx}
f ′ ( x ) = d x d y 微商
①向量微分
x
∈
R
n
×
1
\textbf{x}\in R^{n\times 1}
x ∈ R n × 1
d
y
=
∑
i
=
1
n
d
y
d
x
i
d
x
i
=
(
d
y
d
x
)
T
d
x
dy = \sum_{i=1}^n \frac{dy}{dx_i} dx_i=(\frac{dy}{dx})^T dx
d y = i = 1 ∑ n d x i d y d x i = ( d x d y ) T d x ②矩阵微分
【例如】：假设
A
∈
R
3
×
2
,
B
∈
R
2
×
3
,
A
B
∈
R
3
×
3
A\in R^{3\times 2},B\in R^{2\times 3},AB\in R^{3\times 3}
A ∈ R 3 × 2 , B ∈ R 2 × 3 , A B ∈ R 3 × 3
n
=
3
,
m
=
2
n=3,m=2
n = 3 , m = 2
A
=
[
a
11
a
12
a
21
a
22
a
31
a
32
]
B
=
[
b
11
b
12
b
13
b
21
b
22
b
23
]
\begin{gathered} A=\begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \\ a_{31} & a_{32} \\ \end{bmatrix} B=\begin{bmatrix} b_{11} & b_{12} & b_{13}\\ b_{21} & b_{22} & b_{23}\\ \end{bmatrix} \end{gathered}
A = ⎣ ⎡ a 1 1 a 2 1 a 3 1 a 1 2 a 2 2 a 3 2 ⎦ ⎤ B = [ b 1 1 b 2 1 b 1 2 b 2 2 b 1 3 b 2 3 ]
A
B
=
[
a
11
b
11
+
a
12
b
21
⋅
⋅
⋅
⋅
⋅
⋅
⋅
⋅
⋅
a
21
b
12
+
a
22
b
22
⋅
⋅
⋅
⋅
⋅
⋅
⋅
⋅
⋅
a
31
b
13
+
a
32
b
23
]
\begin{gathered} A B=\begin{bmatrix} a_{11}b_{11}+a_{12}b_{21} & ··· & ··· \\ ···& a_{21}b_{12}+a_{22}b_{22} & ···\\ ··· & ··· & a_{31}b_{13}+a_{32}b_{23} \\ \end{bmatrix} \end{gathered}
A B = ⎣ ⎡ a 1 1 b 1 1 + a 1 2 b 2 1 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ a 2 1 b 1 2 + a 2 2 b 2 2 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ a 3 1 b 1 3 + a 3 2 b 2 3 ⎦ ⎤
A
,
B
A,B
A , B
A
A
A
B
T
B^T
B T
t
r
(
A
B
)
=
∑
i
=
1
n
(
A
B
)
i
i
=
∑
i
=
1
n
∑
j
=
1
m
A
i
j
B
j
i
=
∑
j
=
1
m
∑
i
=
1
n
B
j
i
A
i
j
tr(AB)=\sum_{i=1}^n(AB)_{ii}=\sum_{i=1}^n\sum_{j=1}^m A_{ij}B_{ji}=\sum_{j=1}^m\sum_{i=1}^n B_{ji}A_{ij}
t r ( A B ) = i = 1 ∑ n ( A B ) i i = i = 1 ∑ n j = 1 ∑ m A i j B j i = j = 1 ∑ m i = 1 ∑ n B j i A i j
X
∈
R
n
×
m
X\in R^{n\times m}
X ∈ R n × m
d
y
=
∑
i
=
1
n
∑
j
=
1
m
d
y
X
i
j
d
X
i
j
=
∑
i
,
j
=
1
(
d
y
X
j
i
)
T
d
X
i
j
=
t
r
[
(
d
y
X
)
T
d
X
]
dy = \sum_{i=1}^n\sum_{j=1}^m \frac{dy}{X_{ij}} dX_{ij} = \sum_{i,j=1}(\frac{dy}{X_{ji}})^T dX_{ij} =tr[(\frac{dy}{X})^T dX]
d y = i = 1 ∑ n j = 1 ∑ m X i j d y d X i j = i , j = 1 ∑ ( X j i d y ) T d X i j = t r [ ( X d y ) T d X ]
微分加减法：
d
(
X
±
Y
)
=
d
(
X
)
±
d
(
Y
)
d(X\pm Y)=d(X)\pm d(Y)
d ( X ± Y ) = d ( X ) ± d ( Y )
微分乘法：
d
(
X
Y
)
=
d
(
X
)
⋅
Y
+
X
⋅
d
(
Y
)
d(XY) = d(X)\cdot Y + X\cdot d(Y)
d ( X Y ) = d ( X ) ⋅ Y + X ⋅ d ( Y )
微分转置：
d
(
X
T
)
=
[
d
(
X
)
]
T
d(X^T)=[d(X)]^T
d ( X T ) = [ d ( X ) ] T
微分的迹：
d
[
t
r
(
X
)
]
=
t
r
[
d
(
X
)
]
d[tr(X)]=tr[d(X)]
d [ t r ( X ) ] = t r [ d ( X ) ]
逆矩阵微分：
d
(
X
−
1
)
=
−
X
−
1
d
(
X
)
X
−
1
d(X^{-1})=-X^{-1}d(X)X^{-1}
d ( X − 1 ) = − X − 1 d ( X ) X − 1
【所需公式】：
标量
x
x
x
t
r
(
x
)
=
x
tr(x) = x
t r ( x ) = x
转置不变性：
t
r
(
A
T
)
=
t
r
(
A
)
tr(A^T)=tr(A)
t r ( A T ) = t r ( A )
交换律：
t
r
(
A
B
)
=
t
r
(
B
T
A
T
)
=
t
r
(
B
A
)
tr(AB)=tr(B^TA^T)=tr(BA)
t r ( A B ) = t r ( B T A T ) = t r ( B A )
A
,
B
T
A,B^T
A , B T
加减法：
t
r
(
A
±
B
)
=
t
r
(
A
)
±
t
r
(
B
)
tr(A\pm B)=tr(A) \pm tr(B)
t r ( A ± B ) = t r ( A ) ± t r ( B )
矩阵逐元素乘法和迹交换：
t
r
[
(
A
⊙
B
)
T
C
]
=
t
r
[
A
T
(
B
⊙
C
)
]
tr[(A\odot B)^T C]=tr[A^T (B\odot C)]
t r [ ( A ⊙ B ) T C ] = t r [ A T ( B ⊙ C ) ]
A
,
B
,
C
A,B,C
A , B , C
A
⊙
B
A \odot B
A ⊙ B 逐元素相乘 的哈达马积 (
H
a
d
a
m
a
r
d
p
r
o
d
u
c
t
Hadamard\; product
H a d a m a r d p r o d u c t
A
⊙
B
=
[
1
2
3
4
]
⊙
[
1
2
3
4
]
=
[
1
⋅
1
2
⋅
2
3
⋅
3
4
⋅
4
]
=
[
1
4
9
16
]
\begin{gathered} A \odot B= \begin{bmatrix} 1 & 2 \\ 3 & 4 \\ \end{bmatrix}\odot \begin{bmatrix} 1 & 2 \\ 3 & 4 \\ \end{bmatrix} = \begin{bmatrix} 1\cdot1 & 2\cdot 2 \\ 3\cdot 3 & 4\cdot 4 \\ \end{bmatrix} = \begin{bmatrix} 1 & 4 \\ 9 & 16 \\ \end{bmatrix} \end{gathered}
A ⊙ B = [ 1 3 2 4 ] ⊙ [ 1 3 2 4 ] = [ 1 ⋅ 1 3 ⋅ 3 2 ⋅ 2 4 ⋅ 4 ] = [ 1 9 4 1 6 ]
d
(
e
X
b
)
=
e
X
b
⊙
d
(
X
b
)
\begin{gathered} d(e^{Xb}) =e^{Xb} \odot d(Xb) \end{gathered}
d ( e X b ) = e X b ⊙ d ( X b )
【求导步骤】:如
y
=
a
T
X
b
y=a^TXb
y = a T X b
d
y
d
X
\frac{dy}{dX}
d X d y 已知
d
y
=
t
r
[
(
d
y
X
)
T
d
X
]
dy =tr[(\frac{dy}{X})^T dX]
d y = t r [ ( X d y ) T d X ]
① 求微分
d
y
=
a
T
d
(
X
)
b
dy = a^T d(X)b
d y = a T d ( X ) b ② 加迹函数以将
d
x
dx
d x 调整矩阵乘积顺序
d
y
=
t
r
(
d
y
)
=
t
r
(
a
T
d
(
X
)
b
)
=
t
r
(
b
a
T
d
X
)
dy = tr(dy)=tr(a^T d(X)b) = tr(ba^T dX)
d y = t r ( d y ) = t r ( a T d ( X ) b ) = t r ( b a T d X )
d
y
d
X
=
(
b
a
T
)
T
=
a
b
T
\frac{dy}{dX} = (ba^T)^T=ab^T
d X d y = ( b a T ) T = a b T
y
=
H
T
L
H
y=H^TLH
y = H T L H
∂
H
T
L
H
∂
H
\frac{\partial H^TLH}{\partial H}
∂ H ∂ H T L H ① 求微分
d
y
=
d
(
H
T
)
L
H
+
H
T
L
d
(
H
)
=
d
(
H
)
T
L
H
+
H
T
L
d
(
H
)
dy = d(H^T)LH+H^TLd(H)= d(H)^TLH+H^TLd(H)
d y = d ( H T ) L H + H T L d ( H ) = d ( H ) T L H + H T L d ( H ) ② 加迹函数以将
d
x
dx
d x
d
y
=
t
r
(
d
y
)
=
t
r
[
d
(
H
)
T
L
H
+
H
T
L
d
(
H
)
]
=
t
r
[
d
(
H
)
T
L
H
]
+
t
r
[
H
T
L
d
(
H
)
]
=
t
r
[
H
T
L
T
d
(
H
)
]
+
t
r
[
H
T
L
d
(
H
)
]
=
t
r
[
H
T
(
L
T
+
L
)
d
H
]
\begin{aligned} dy = tr(dy) & = tr[d(H)^TLH+H^TLd(H)]=tr[d(H)^TLH]+tr[H^TLd(H)] \\ & = tr[H^TL^Td(H)]+tr[H^TLd(H)] \\ & = tr[H^T(L^T+L)dH] \\ \end{aligned}
d y = t r ( d y ) = t r [ d ( H ) T L H + H T L d ( H ) ] = t r [ d ( H ) T L H ] + t r [ H T L d ( H ) ] = t r [ H T L T d ( H ) ] + t r [ H T L d ( H ) ] = t r [ H T ( L T + L ) d H ]
d
y
d
H
=
∂
H
T
L
H
∂
H
=
(
L
T
+
L
)
T
H
=
(
L
+
L
T
)
H
\frac{d y}{d H} = \frac{\partial H^TLH}{\partial H} = (L^T+L)^TH=(L+L^T)H
d H d y = ∂ H ∂ H T L H = ( L T + L ) T H = ( L + L T ) H
矩阵的链式法则详情求导见机器学习中的矩阵向量求导四
【参考文献】[1] 刘建平.机器学习中的矩阵向量求导[一] [2] 刘建平.机器学习中的矩阵向量求导[二] [3] 刘建平.机器学习中的矩阵向量求导[三] [4] 刘建平.机器学习中的矩阵向量求导[四] [5] 矩阵求导术（上）[知乎] 【排版真心不敢恭维】[6] 矩阵求导术（下）[知乎] 【排版真心不敢恭维】
