Coursera机器学习笔记 第5周 第九章 神经网络 相关数学公式证明

版权声明：本文为博主原创文章，如有转载请标注来源。 https://blog.csdn.net/shengchaohua163/article/details/79747381

神经网络 相关数学公式证明

0. Coursera机器学习神经网络的其他笔记：

1. Coursera机器学习笔记 第4周 第八章 神经网络：表述（一）
2. Coursera机器学习笔记 第4周 第八章 神经网络：表述（二）
3. Coursera机器学习笔记 第5周 第九章 神经网络：学习（一）
4. Coursera机器学习笔记 第5周 第九章 神经网络：学习（二）

1. 神经网络的回顾

我们仍然沿用上图中的神经网络结构。该网络共有四层，输入层有3个神经元，第二、三层均有5个神经元，输出层有4个神经元（偏置单元未画出），所以权重矩阵$\Theta^{(1)}, \Theta^{(2)}, \Theta^{(3)}$的大小分别是$[5 \times 4], [5 \times 6], [4 \times 6]$。权重矩阵如下：

$$\Theta^{(1)}=\begin{bmatrix} \Theta^{(1)}_{10} & ... & \Theta^{(1)}_{13}\\ ... & ... & ...\\ \Theta^{(1)}_{50} & ... & \Theta^{(1)}_{53} \end{bmatrix} \Theta^{(2)}=\begin{bmatrix} \Theta^{(2)}_{10} & ... & \Theta^{(2)}_{15}\\ ... & ... & ...\\ \Theta^{(2)}_{50} & ... & \Theta^{(2)}_{55} \end{bmatrix} \Theta^{(3)}=\begin{bmatrix} \Theta^{(3)}_{10} & ... & \Theta^{(1)}_{15}\\ ... & ... & ...\\ \Theta^{(3)}_{40} & ... & \Theta^{(1)}_{45} \end{bmatrix}$$
为了和代价函数的形式统一，我们用 j 和 i 分别表示权重矩阵的行和列。注意：j=1 代表矩阵的第 1 行，i=1 代表矩阵的第 2 列（第 1 列相当于线性回归中的$\theta_0$）。

神经网络的一般化代价函数为：

$$\begin{gather*}J(\Theta) = - \frac{1}{m} \sum_{t=1}^m\sum_{k=1}^K \left[ y^{(t)}_k \ \log (h_\Theta (x^{(t)})_k) + (1 - y^{(t)}_k)\ \log (1 - h_\Theta(x^{(t)})_k)\right] + \frac{\lambda}{2m}\sum_{l=1}^{L-1} \sum_{i=1}^{s_l} \sum_{j=1}^{s_l+1} ( \Theta_{j,i}^{(l)})^2\end{gather*}$$

我们只用一条数据(x,y)，并且忽略正则化，则代价函数(K=4)为：

$$\begin{gather*}Cost(x) = \sum_{k=1}^K Cost(x)_k \end{gather*} \\ Cost(x)_k = - \left[ y_k \ \log (h_\Theta (x)_k) + (1 - y_k)\ \log (1 - h_\Theta(x)_k)\right]$$

2. 证明$\dfrac{\partial Cost(x)}{\partial \Theta^{(3)}}$

针对该神经网络，声明一些事实：

• $h_\Theta (x)$就是$a^{(4)}$，即$h_\Theta (x)=a^{(4)}$
• $a^{(4)} = g(z^{(4)})$，且$\dfrac{\partial a^{(4)}}{\partial z^{(4)}} = a^{(4)} (1 - a^{(4)})$
• $z^{(4)}_j = \sum^{5}_{i=0}\Theta^{(3)}_{ji}a^{(3)}_i$，其中 $1 \leqslant j \leqslant 4$
• j=1 代表矩阵的第 1 行，i=1 代表矩阵的第 2 列（第 1 列相当于线性回归中的$\theta_0$）

举例：

$$\begin{align*}\dfrac{\partial Cost(x)}{\partial \Theta_{10}^{(3)}} &= \dfrac{\partial \sum^K_{k=1}Cost(x)_k}{\partial \Theta_{10}^{(3)}} \\ &= \sum^K_{k=1} \dfrac{\partial Cost(x)_k}{\partial \Theta_{10}^{(3)}} \\ &= \dfrac{\partial Cost(x)_1}{\partial \Theta_{10}^{(3)}} + 0 + 0 + 0 \\ &= \dfrac{\partial Cost(x)_1}{\partial h_{\Theta}(x)_1} \times \dfrac{\partial h_{\Theta}(x)_1}{\partial z^{(4)}_1} \times \dfrac{\partial z^{(4)}_1}{\partial \Theta_{10}^{(3)}} \\ &= -[y_1\dfrac{1}{h_{\Theta}(x)_1}+(1-y_1)\dfrac{-1}{1-h_{\Theta}(x)_1}] \times h_{\Theta}(x)_1(1-h_{\Theta}(x)_1) \times a^{(3)}_0 \\ &= [h_{\Theta}(x)_1 - y_1]a^{(3)}_0 \\ &= [a^{(4)}_1 - y_1]a^{(3)}_0 \end{align*}$$ $$\begin{align*}\dfrac{\partial Cost(x)}{\partial \Theta_{20}^{(3)}} &= 0 + \dfrac{\partial Cost(x)_2}{\partial \Theta_{20}^{(3)}} + 0 + 0 \\ &= \dfrac{\partial Cost(x)_2}{\partial h_{\Theta}(x)_2} \times \dfrac{\partial h_{\Theta}(x)_2}{\partial z^{(4)}_2} \times \dfrac{\partial z^{(4)}_2}{\partial \Theta_{20}^{(3)}} \\ &= -[y_2\dfrac{1}{h_{\Theta}(x)_2}+(1-y_2)\dfrac{-1}{1-h_{\Theta}(x)_2}] \times h_{\Theta}(x)_2(1-h_{\Theta}(x)_2) \times a^{(3)}_0 \\ &= [h_{\Theta}(x)_2 - y_2]a^{(3)}_0 \end{align*}$$ $$\begin{align*}\dfrac{\partial Cost(x)}{\partial \Theta_{21}^{(3)}} &= 0 + \dfrac{\partial Cost(x)_2}{\partial \Theta_{21}^{(3)}} + 0 + 0 \\ &= \dfrac{\partial Cost(x)_2}{\partial h_{\Theta}(x)_2} \times \dfrac{\partial h_{\Theta}(x)_2}{\partial z^{(4)}_2} \times \dfrac{\partial z^{(4)}_2}{\partial \Theta_{21}^{(3)}} \\ &= -[y_2\dfrac{1}{h_{\Theta}(x)_2}+(1-y_2)\dfrac{-1}{1-h_{\Theta}(x)_2}] \times h_{\Theta}(x)_2(1-h_{\Theta}(x)_2) \times a^{(3)}_1 \\ &= [h_{\Theta}(x)_2 - y_2]a^{(3)}_1 \end{align*}$$
综上： $$\begin{align*}\dfrac{\partial Cost(x)}{\partial \Theta_{ji}^{(3)}} &= \dfrac{\partial Cost(x)_j}{\partial h_{\Theta}(x)_j} \times \dfrac{\partial h_{\Theta}(x)_j}{\partial z^{(4)}_j} \times \dfrac{\partial z^{(4)}_j}{\partial \Theta_{ji}^{(3)}} \\ &= -[y_j\dfrac{1}{h_{\Theta}(x)_j}+(1-y_j)\dfrac{-1}{1-h_{\Theta}(x)_j}] \times h_{\Theta}(x)_j(1-h_{\Theta}(x)_j) \times a^{(3)}_i \\ &= [h_{\Theta}(x)_j - y_j]（a^{(3)}_i \end{align*}$$
引入 $\color{Red}{\delta^{(4)} = a^{(4)} - y}$，再加上 $h_\Theta (x) =a^{(4)}$ ，所以 $\dfrac{\partial Cost(x)}{\partial \Theta_{ji}^{(3)}} =\delta^{(4)}_j a^{(3)}_i$
把矩阵下标去掉，公式变为： $\dfrac{\partial Cost(x)}{\partial \Theta^{(3)}} =\color{Red}{\delta^{(4)}}(a^{(3)})^T$

3. 证明$\dfrac{\partial Cost(x)}{\partial \Theta^{(2)}}，\dfrac{\partial Cost(x)}{\partial \Theta^{(1)}}$

举例：

$$\begin{align*}\dfrac{\partial Cost(x)}{\partial \Theta_{10}^{(2)}} &= \sum^{K}_{k=1}\dfrac{\partial Cost(x)_k}{\partial \Theta_{10}^{(2)}} \\ &= \sum^{K}_{k=1} \left[\dfrac{\partial Cost(x)_k}{\partial h_{\Theta}(x)_k} \times \dfrac{\partial h_{\Theta}(x)_k}{\partial z^{(4)}_k} \times \dfrac{\partial z^{(4)}_k}{\partial a^{(3)}_1} \times \dfrac{\partial a^{(3)}_1}{\partial z^{(3)}_1} \times \dfrac{\partial z^{(3)}_1}{\partial \Theta^{(2)}_{10}}\right] \\ &= \sum^{K}_{k=1} \left[(h_{\Theta}(x)_k-y_k) \times \Theta^{(3)}_{k1} \times a^{(3)}_1(1-a^{(3)}_1) \times a^{(2)}_0\right]\end{align*}$$ $$\begin{align*}\dfrac{\partial Cost(x)}{\partial \Theta_{20}^{(2)}} &= \sum^{K}_{k=1}\dfrac{\partial Cost(x)_k}{\partial \Theta_{20}^{(2)}} \\ &= \sum^{K}_{k=1} \left[\dfrac{\partial Cost(x)_k}{\partial h_{\Theta}(x)_k} \times \dfrac{\partial h_{\Theta}(x)_k}{\partial z^{(4)}_k} \times \dfrac{\partial z^{(4)}_k}{\partial a^{(3)}_2} \times \dfrac{\partial a^{(3)}_2}{\partial z^{(3)}_2} \times \dfrac{\partial z^{(3)}_2}{\partial \Theta^{(2)}_{20}}\right] \\ &= \sum^{K}_{k=1} \left[(h_{\Theta}(x)_k-y_k) \times \Theta^{(3)}_{k2} \times a^{(3)}_2(1-a^{(3)}_2) \times a^{(2)}_0\right]\end{align*}$$ $$\begin{align*}\dfrac{\partial Cost(x)}{\partial \Theta_{21}^{(2)}} &= \sum^{K}_{k=1}\dfrac{\partial Cost(x)_k}{\partial \Theta_{21}^{(2)}} \\ &= \sum^{K}_{k=1} \left[\dfrac{\partial Cost(x)_k}{\partial h_{\Theta}(x)_k} \times \dfrac{\partial h_{\Theta}(x)_k}{\partial z^{(4)}_k} \times \dfrac{\partial z^{(4)}_k}{\partial a^{(3)}_2} \times \dfrac{\partial a^{(3)}_2}{\partial z^{(3)}_2} \times \dfrac{\partial z^{(3)}_2}{\partial \Theta^{(2)}_{21}}\right] \\ &= \sum^{K}_{k=1} \left[(h_{\Theta}(x)_k-y_k) \times \Theta^{(3)}_{k2} \times a^{(3)}_2(1-a^{(3)}_2) \times a^{(2)}_1\right]\end{align*}$$ $$\begin{align*}\dfrac{\partial Cost(x)}{\partial \Theta_{32}^{(2)}} &= \sum^{K}_{k=1}\dfrac{\partial Cost(x)_k}{\partial \Theta_{32}^{(2)}} \\ &= \sum^{K}_{k=1} \left[\dfrac{\partial Cost(x)_k}{\partial h_{\Theta}(x)_k} \times \dfrac{\partial h_{\Theta}(x)_k}{\partial z^{(4)}_k} \times \dfrac{\partial z^{(4)}_k}{\partial a^{(3)}_3} \times \dfrac{\partial a^{(3)}_3}{\partial z^{(3)}_3} \times \dfrac{\partial z^{(3)}_3}{\partial \Theta^{(2)}_{32}}\right] \\ &= \sum^{K}_{k=1} \left[(h_{\Theta}(x)_k-y_k) \times \Theta^{(3)}_{k3} \times a^{(3)}_3(1-a^{(3)}_3) \times a^{(2)}_2\right]\end{align*}$$

扫描二维码关注公众号，回复： 3437484 查看本文章

记住$\color{Red}{\delta^{(4)} = a^{(4)} - y, h_\Theta (x) =a^{(4)} }$。根据前面的例子把公式进行一般化（其中$(\Theta^{(3)})^T_{j:}$表示矩阵$((\Theta^{(3)})^T$第 j 行，第一个乘号是矩阵相乘，其他乘号是实数相乘）：

$$\begin{align*}\dfrac{\partial Cost(x)}{\partial \Theta_{ji}^{(2)}} &= \sum^{K}_{k=1} \left[(h_{\Theta}(x)_k-y_k) \times \Theta^{(3)}_{kj} \times a^{(3)}_j(1-a^{(3)}_j) \times a^{(2)}_i\right] \\ &=\sum^{K}_{k=1} \left[\delta^{(4)}_k \times \Theta^{(3)}_{kj}\right] \times a^{(3)}_j(1-a^{(3)}_j) \times a^{(2)}_i \\ &=(\Theta^{(3)})^T_{j:} \times \delta^{(4)} \times a^{(3)}_j(1-a^{(3)}_j) \times a^{(2)}_i\end{align*}$$

把矩阵下标去掉，公式变为（两个乘号表示矩阵相乘，.* 表示对应元素相乘）：

$$\begin{align*}\dfrac{\partial Cost(x)}{\partial \Theta^{(2)}} &=((\Theta^{(3)})^T \times \delta^{(4)}) .*\ a^{(3)} .*\ (1-a^{(3)}) \times a^{(2)}\end{align*}$$

再引入$\color{Red}{\delta^{(3)} = ((\Theta^{(3)})^T \delta^{(4)})\ .*\ a^{(3)}\ .*\ (1 - a^{(3)}) }$，公式变为：

$$\begin{align*}\dfrac{\partial Cost(x)}{\partial \Theta^{(2)}} &=\color{Red}{\delta^{(3)}} a^{(2)}\end{align*}$$

再引入$\color{Red}{\delta^{(2)} = ((\Theta^{(2)})^T \delta^{(3)})\ .*\ a^{(2)}\ .*\ (1 - a^{(2)}) }$，所以

$$\begin{align*}\dfrac{\partial Cost(x)}{\partial \Theta^{(1)}} &=\color{Red}{\delta^{(2)}} a^{(1)}\end{align*}$$

4. 总结

一条数据的反向传播如下：

1. 计算输出层的误差：$\delta^{(4)} = a^{(4)} - y$
2. 计算第三层的误差：$\delta^{(3)} = ((\Theta^{(3)})^T \delta^{(4)})\ .*\ a^{(3)}\ .*\ (1 - a^{(3)})$
3. 计算第二层的误差：$\delta^{(2)} = ((\Theta^{(2)})^T \delta^{(3)})\ .*\ a^{(2)}\ .*\ (1 - a^{(2)})$。第一层是输入变量，不存在误差，所以到第二层即可。
4. 为什么计算每一层的误差 $\delta$ ？因为经过一系列复杂的求导后，我们通过 $\delta$ 可以计算代价函数对每一层权重矩阵的每一个参数的偏导数（无正则化处理或$\lambda=0$）：$\dfrac{\partial}{\partial \Theta_{i,j}^{(l)}}J(\Theta) = \delta^{(l+1)}_i a^{(l)}_j$，其中每个元素都是一个实数！

重点：

• 误差公式：$\color{Red}{\delta^{(l)} = ((\Theta^{(l)})^T \delta^{(l+1)})\ .*\ g'(z^{(l)})}$，其中$\color{Red}{g'(z^{(l)}) = a^{(l)}\ .*\ (1 - a^{(l)})}$
• 偏导数（梯度）公式：$\dfrac{\partial}{\partial \Theta_{i,j}^{(l)}}J(\Theta) = \delta^{(l+1)}_i a^{(l)}_j$
• 偏导数（梯度）公式（矩阵形式）：$\dfrac{\partial}{\partial \Theta^{(l)}}J(\Theta) = \delta^{(l+1)} (a^{(l)})^T$