向量的导数运算和向量叉乘以及点乘的导数运算

目的：最近在写优化代码，需要对函数中的变量求导，以及求得它们的雅克比矩阵。因此用到向量以及矩阵的求导。

一个向量可以表示为如下：$Y=[y_1,y_2,...,y_m{]}^T$
向量导数的基本知识。它分为以下几类：
1)向量$Y=[y_1,y_2,...,y_m{]}^T$对$x$标量求导:
$$\frac{\partial Y}{\partial x}=\left[\begin{array}{c}\frac{\partial y_1}{\partial x}\\ \frac{\partial y_2}{\partial x}\\ ⋮\\ \frac{\partial y_m}{\partial x}\end{array}\right]$$
如果$Y=[y_1,y_2,...,y_m]$是行向量，则求导
$$\frac{\partial Y}{\partial x}=\left[\begin{array}{c}\frac{\partial y_1}{\partial x}\frac{\partial y_2}{\partial x}\dots \frac{\partial y_m}{\partial x}\end{array}\right]$$

2)标量$y$对向量$X=[x_1,x_2,...,x_m{]}^T$求导
$$\frac{\partial y}{\partial X}=\left[\begin{array}{c}\frac{\partial y}{\partial x_1}\\ \frac{\partial y}{\partial x_2}\\ ⋮\\ \frac{\partial y}{\partial x_m}\end{array}\right]$$
如果$X=[x_1,x_2,...,x_m]$为行向量：
$$\frac{\partial y}{\partial X}=\left[\begin{array}{c}\frac{\partial y}{\partial x_1}\frac{\partial y}{\partial x_2}\dots \frac{\partial y}{\partial x_m}\end{array}\right]$$

3)向量$Y=[y_1,y_2,...,y_m{]}^T$对向量$X=[x_1,x_2,...,x_n]$求导
$$\frac{\partial Y}{\partial X}=\left[\begin{array}{c}\frac{\partial y_1}{\partial x_1}\frac{\partial y_1}{\partial x_2}\dots \frac{\partial y_1}{\partial x_n}\\ \frac{\partial y_2}{\partial x_1}\frac{\partial y_2}{\partial x_2}\dots \frac{\partial y_2}{\partial x_n}\\ ⋮\\ \frac{\partial y_m}{\partial x_1}\frac{\partial y_m}{\partial x_2}\dots \frac{\partial y_m}{\partial x_n}\end{array}\right]$$
向量对向量求导也是所谓的雅克比矩阵，它在优化中非常见。

如果是矩阵的话，
如$Y$是矩阵的时候，它的表达：
$$Y=\left[\begin{array}{c}{y}_{11}{y}_{12}\dots y_{1n}\\ y_{21}{y}_{22}\dots y_{2n}\\ ⋮\\ y_{m1}{y}_{m2}\dots y_{mn}\end{array}\right]$$
如$X$是矩阵的时候，它的表达：
$$X=\left[\begin{array}{c}{x}_{11}{x}_{12}\dots x_{1n}\\ x_{21}{x}_{22}\dots x_{2n}\\ ⋮\\ x_{m1}{x}_{m2}\dots x_{mn}\end{array}\right]$$

矩阵的导数有两种，如下
1)矩阵$Y$对标量$x$求导:
$$\frac{\partial Y}{\partial x}=\left[\begin{array}{c}\frac{\partial y_{11}}{\partial x}\frac{\partial y_{12}}{\partial x}\dots \frac{\partial y_{1n}}{\partial x}\\ \frac{\partial y_{21}}{\partial x}\frac{\partial y_{22}}{\partial x}\dots \frac{\partial y_{2n}}{\partial x}\\ ⋮\\ \frac{\partial y_{m1}}{\partial x}\frac{\partial y_{m2}}{\partial x}\dots \frac{\partial y_{mn}}{\partial x}\end{array}\right]$$
2)标量$y$对矩阵$X$求导:
$$\frac{\partial y}{\partial X}=\left[\begin{array}{c}\frac{\partial y}{\partial x_{11}}\frac{\partial y}{\partial x_{12}}\dots \frac{\partial y}{\partial x_{1n}}\\ \frac{\partial y}{\partial x_{21}}\frac{\partial y}{\partial x_{22}}\dots \frac{\partial y}{\partial x_{2n}}\\ ⋮\\ \frac{\partial y}{\partial x_{m1}}\frac{\partial y}{\partial x_{m2}}\dots \frac{\partial y}{\partial x_{mn}}\end{array}\right]$$
这是基本的向量的导数定义。基于这些定义以及一些基本的运算法则，得到一些组合的公式。在几何算法的编程中非常有用。

公式中的向量求导，在一般公式中会多个向量以及向量依赖，因此，在求导数的时候希望它能满足标量求导的链式法则。
假设向量相互依赖的关系为：$U->V->W$
则偏导数为：
$$\frac{\partial W}{\partial U}=\frac{\partial W}{\partial V}\frac{\partial V}{\partial U}$$

证明：只需要拆开逐一对元素求导得到：
$$\frac{\partial w_i}{\partial u_j}=\sum _k\frac{\partial w_i}{\partial v_k}\frac{\partial v_k}{\partial u_j}=\frac{\partial w_i}{\partial V}\frac{\partial V}{\partial u_j}$$
由此可见$\frac{\partial w_i}{\partial u_j}$是等于矩阵$\frac{\partial W}{\partial V}$第$i$行和矩阵$\frac{\partial V}{\partial U}$的第$j$列的内积，这是矩阵的乘法定义。
它很容易能推广到多层中间变量的情景。

在变量中遇到的情况是常常公式为$F$为一个实数，中间变量都是向量的时候，它的依赖为：
$X->V->U->f$
根据雅克比矩阵的传递性可以得到如下：
$$\frac{\partial F}{\partial X}=\frac{\partial F}{\partial U}\frac{\partial U}{\partial V}\frac{\partial V}{\partial X}$$
因为$f$为标量，因此它写成如下形式：
$$\frac{\partial f}{\partial X^T}=\frac{\partial f}{\partial U^T}\frac{\partial U}{\partial V}\frac{\partial V}{\partial X}$$
为了便于计算，上述需要转为行向量$U^T$,$X^T$计算。这个非常重要。

下面介绍向量求倒数的时候遇到的常用的运算公式，它们有以下两类

1）两向量$U$,$V$（列向量）点积的结果对$W$求导:
$$\frac{\partial (U^{T}V)}{\partial W}=(\frac{\partial U}{\partial W}{)}^{T}V+(\frac{\partial V}{\partial W}{)}^{T}U(4)$$
点积的导数公式证明后续补上。
证明：假设$U=\left[\begin{array}{c}{u}_0\\ u_1\\ u_3\end{array}\right]$ 和$V=\left[\begin{array}{c}{v}_0\\ v_1\\ v_3\end{array}\right]$ ，它们为三维向量。得到点乘为$f=U^{T}V$，它是一个标量为：$f=u_0{v}_0+u_1{v}_1+u_2{v}_2$，然后求它对$W$的导数

$$\begin{gathered} \frac{\partial f}{\partial W}=\frac{\partial (u_0{v}_0+u_1{v}_1+u_2{v}_2)}{\partial W} \\ =\frac{\partial u_0}{\partial W}{v}_0+\frac{\partial v_0}{\partial W}{u}_0+\frac{\partial u_1}{\partial W}{v}_1+\frac{\partial v_1}{\partial W}{u}_1+\frac{\partial u_2}{\partial W}{v}_2+\frac{\partial v_2}{\partial W}{u}_2 \\ =(\frac{\partial u_0}{\partial W}{v}_0+\frac{\partial u_1}{\partial W}{v}_1+\frac{\partial u_2}{\partial W}{v}_2)+(\frac{\partial v_0}{\partial W}{u}_0+\frac{\partial v_1}{\partial W}{u}_1+\frac{\partial v_2}{\partial W}{u}_2) \\ =(\frac{\partial U}{\partial W}{)}^{T}V+(\frac{\partial V}{\partial W}{)}^{T}U \end{gathered}$$

它可以推广到其它的维度。证明完毕。

如果$W$是标量其实直接代入$(4)$即可。但是如果$W$为向量，在计算中，一般都是把$W$变成行向量。因为定义jacobi矩阵是，列向量对行向量就行求导。因此$W$可以表示为$W^T$(行向量)，所以$(4)$，写成：
$$\frac{\partial (U^{T}V)}{\partial W^T}=(\frac{\partial U}{\partial W^T}{)}^{T}V+(\frac{\partial V}{\partial W^T}{)}^{T}U$$

2）两个向量$U$,$V$ (列向量)叉积的结果对$W$求导:
$$\frac{\partial (U\times V)}{\partial W}=-Skew(V)(\frac{\partial U}{\partial W})+Skew(U)(\frac{\partial V}{\partial W})(5)$$
其中
$$Skew(U)=\left[\begin{array}{c}0-U_3{U}_2\\ U_{3}0-U_1\\ -U_2{U}_{1}0\end{array}\right]$$
其中$Skew(V)$是将叉乘转化为点积的矩阵。它非常容易证明，因为它就是矩阵展开即可。
对于多个向量叉乘的时候，需要对公式进行转化。叉乘满足分配率。
$$\frac{\partial (U\times V)}{\partial W}=(\frac{\partial U}{\partial W})\times V+U\times (\frac{\partial V}{\partial W})(6)$$
证明后续再补上。(5)和(6)两者的公式是想通的。只是表达形式不同。它们的转化后面再补上。

证明：假设$U=\left[\begin{array}{c}{u}_0\\ u_1\\ u_3\end{array}\right]$ 和$V=\left[\begin{array}{c}{v}_0\\ v_1\\ v_3\end{array}\right]$ ，它们为三维向量。

$$\begin{gathered} U\times V=\left[\begin{array}{c}ijk\\ u_0{u}_1{u}_2\\ v_0{v}_1{v}_2\end{array}\right] \\ =(u_1{v}_2-u_1{v}_2)i+(u_2{v}_0-u_0{v}_2)j+(u_0{v}_1-u_1{v}_0)k \end{gathered}$$

它是一个向量，因此展开后，它的表达为如下：

$$U\times V=\left[\begin{array}{c}(u_1{v}_2-u_2{v}_1)\\ (u_2{v}_0-u_0{v}_2)\\ (u_0{v}_1-u_1{v}_0)\end{array}\right]$$

展开后得到如下：

$$\begin{gathered} \frac{\partial (U\times V)}{\partial W}=\left[\begin{array}{c}\frac{\partial (u_1{v}_2-u_2{v}_1)}{\partial W}\\ \frac{\partial (u_2{v}_0-u_0{v}_2)}{\partial W}\\ \frac{\partial (u_0{v}_1-u_1{v}_0)}{\partial W}\end{array}\right]=\frac{\partial (u_1{v}_2-u_2{v}_1)}{\partial W}I+\frac{\partial (u_2{v}_0-u_0{v}_2)}{\partial W}J+\frac{\partial (u_0{v}_1-u_1{v}_0)}{\partial W}K \\ =(\frac{\partial u_1}{\partial W}\ast v_2+\frac{\partial v_2}{\partial W}\ast u_1-\frac{\partial u_2}{\partial W}\ast v_1-\frac{\partial v_1}{\partial W}\ast u_2)I+(\frac{\partial u_2}{\partial W}\ast v_0+\frac{\partial v_0}{\partial W}\ast u_2-\frac{\partial u_0}{\partial W}\ast v_2-\frac{\partial v_2}{\partial W}\ast u_0)J+(\frac{\partial u_0}{\partial W}\ast v_1+\frac{\partial v_1}{\partial W}\ast u_0-\frac{\partial u_1}{\partial W}\ast v_0-\frac{\partial v_0}{\partial W}\ast u_1)K \\ =[(\frac{\partial u_1}{\partial W}\ast v_2-\frac{\partial u_2}{\partial W}\ast v_1)I+(\frac{\partial u_2}{\partial W}\ast v_0-\frac{\partial u_0}{\partial W}\ast v_2)J+(\frac{\partial u_0}{\partial W}\ast v_1-\frac{\partial u_1}{\partial W}\ast v_0)K]+[(\frac{\partial v_2}{\partial W}\ast u_1-\frac{\partial v_1}{\partial W}\ast u_2)I+(\frac{\partial v_0}{\partial W}\ast u_2-\frac{\partial v_2}{\partial W}\ast u_0)J+(\frac{\partial v_1}{\partial W}\ast u_0-\frac{\partial v_0}{\partial W}\ast u_1)K] \\ =(\frac{\partial U}{\partial W})\times V-(\frac{\partial V}{\partial W})\times U=-V\times (\frac{\partial U}{\partial W})+U\times (\frac{\partial V}{\partial W})=-Skew(V)(\frac{\partial U}{\partial W})+Skew(U)(\frac{\partial V}{\partial W}) \end{gathered}$$

其中的假设$a,b$为向量，易得如下
$$a\times b=-b\times a$$

从三维可以拓展到多维向量中。证明完毕