概念

反向传播（Back Propagation, BP）算法是使用梯度下降法相关的算法来优化一个神经网络时计算每一层梯度的方法，主要使用了多元函数的链式法则：

已知多元函数 $u=g(y_1,y_2,...,y_m)$ ，且 $y_i=f_i(x)$ ，所有函数都可微，则

$\frac{\partial u}{\partial x} = \sum_{i = 1}^{m} \frac{\partial u}{\partial y_{i}} \frac{\partial y_{i}}{\partial x}$ $\frac{\partial u}{\partial x}=\sum_{i=1}^{m}\frac{\partial u}{\partial y_i}\frac{\partial y_i}{\partial x}$

公式推导

1、模型

不失一般性，我们考虑以下4层结构的神经网络（全连接）：
这里写图片描述

2、符号说明

符号	含义
$n_l$	网络层数
$y_j$	输出层第 $j$ 类标签
$S_l$	第 $l$ 层神经元个数（不包括偏置）
$g(x)$	激活函数
$w_{ij}^{(l)}$	第 $l$ 层第 $j$ 个单元与第 $l+1$ 层第 $i$ 个单元之间的链接参数
$b_i^{(l)}$	第 $l$ 层的偏置与第 $l+1$ 层第 $i$ 个单元之间的链接参数
$z_i^{(l)}$	第 $l$ 层第 $i$ 个单元的输入（加权和，包括偏置）
$a_i^{(l)}$	第 $l$ 层第 $i$ 个单元的输出（激活函数的值）
$\delta_i^{(l)}$	第 $l$ 层第 $i$ 个单元的输入的偏导（或称为灵敏度、残差）
$J(\theta)$	代价函数

3、符号定义

\begin{aligned} z_{i}^{(l)} & = b_{i}^{(l - 1)} + \sum_{j = 1}^{S_{l - 1}} w_{i j}^{(l - 1)} a_{j}^{(l - 1)} \\ a_{i}^{(l)} & = g (z_{i}^{(l)}) \\ J (θ) & = \frac{1}{2} \sum_{j = 1}^{S_{l}} (y_{j} - a_{j}^{(l)})^{2} \\ δ_{i}^{(l)} & = \frac{\partial J (θ)}{\partial z_{i}^{(l)}} \end{aligned}

$\begin{align*} z_i^{(l)}&=b_i^{(l-1)}+\sum_{j=1}^{S_{l-1}}w_{ij}^{(l-1)}a_j^{(l-1)} \\ a_i^{(l)}&=g(z_i^{(l)}) \\ J(\theta)&=\frac{1}{2}\sum_{j=1}^{S_l}(y_j-a_j^{(l)})^2 \\ \delta_i^{(l)}&=\frac{\partial J(\theta)}{\partial z_i^{(l)}} \end{align*}$

4、推导过程

\begin{aligned} δ_{i}^{(n_{l})} & = \frac{\partial J (θ)}{\partial z_{i}^{(n_{l})}} \\ = \frac{1}{2} \frac{\partial}{\partial z_{i}^{(n_{l})}} \sum_{j = 1}^{S_{n_{l}}} (y_{j} - a_{j}^{(n_{l})})^{2} \\ = \frac{1}{2} \frac{\partial}{\partial z_{i}^{(n_{l})}} \sum_{j = 1}^{S_{n_{l}}} (y_{j} - g (z_{j}^{(n_{l})}))^{2} \\ = \frac{1}{2} \frac{\partial}{\partial z_{i}^{(n_{l})}} (y_{j} - g (z_{i}^{(n_{l})}))^{2} \\ = - (y_{i} - a_{i}^{(n_{l})}) g^{'} (z_{i}^{(n_{l})}) \\ δ_{i}^{(l)} & = \frac{\partial J (θ)}{\partial z_{i}^{(l)}} \\ = \sum_{j = 1}^{S_{l + 1}} \frac{\partial J (θ)}{\partial z_{j}^{(l + 1)}} \frac{\partial z_{j}^{(l + 1)}}{\partial z_{i}^{(l)}} \\ = \sum_{j = 1}^{S_{l + 1}} δ_{j}^{(l + 1)} \frac{\partial z_{j}^{(l + 1)}}{\partial z_{i}^{(l)}} \\ = \sum_{j = 1}^{S_{l + 1}} δ_{j}^{(l + 1)} \frac{\partial}{\partial z_{i}^{(l)}} (b_{j}^{(l)} + \sum_{k = 1}^{S_{l}} w_{j k}^{(l)} a_{k}^{(l)}) \\ = \sum_{j = 1}^{S_{l + 1}} δ_{j}^{(l + 1)} \frac{\partial}{\partial z_{i}^{(l)}} (b_{j}^{(l)} + \sum_{k = 1}^{S_{l}} w_{j k}^{(l)} g (z_{k}^{(l)})) \\ = \sum_{j = 1}^{S_{l + 1}} δ_{j}^{(l + 1)} \frac{\partial}{\partial z_{i}^{(l)}} (w_{j i}^{(l)} g (z_{i}^{(l)})) \\ = \sum_{j = 1}^{S_{l + 1}} δ_{j}^{(l + 1)} w_{j i}^{(l)} g^{'} (z_{i}^{(l)}) \\ = g^{'} (z_{i}^{(l)}) \sum_{j = 1}^{S_{l + 1}} δ_{j}^{(l + 1)} w_{j i}^{(l)} \\ \frac{\partial J (θ)}{\partial w_{i j}^{(l)}} & = \frac{\partial J (θ)}{\partial z_{i}^{(l + 1)}} \frac{\partial z_{i}^{(l + 1)}}{\partial w_{i j}^{(l)}} \\ = δ_{i}^{(l + 1)} \frac{\partial z_{i}^{(l + 1)}}{\partial w_{i j}^{(l)}} \\ = δ_{i}^{(l + 1)} \frac{\partial}{\partial w_{i j}^{(l)}} (b_{i}^{(l)} + \sum_{k = 1}^{S_{l}} w_{i k}^{(l)} a_{k}^{(l)}) \\ = δ_{i}^{(l + 1)} a_{j}^{(l)} \\ \frac{\partial J (θ)}{\partial b_{i}^{(l)}} & = δ_{i}^{(l + 1)} \frac{\partial}{\partial b_{i}^{(l)}} (b_{i}^{(l)} + \sum_{k = 1}^{S_{l}} w_{i k}^{(l)} a_{k}^{(l)}) \\ = δ_{i}^{(l + 1)} \end{aligned}

$\begin{align*} \delta_i^{(n_l)}&=\frac{\partial J(\theta)}{\partial z_i^{(n_l)}}\\ &=\frac{1}{2}\frac{\partial}{\partial z_i^{(n_l)}}\sum_{j=1}^{S_{n_l}}(y_j-a_j^{(n_l)})^2 \\ &=\frac{1}{2}\frac{\partial}{\partial z_i^{(n_l)}}\sum_{j=1}^{S_{n_l}}(y_j-g(z_j^{(n_l)}))^2 \\ &=\frac{1}{2}\frac{\partial}{\partial z_i^{(n_l)}}(y_j-g(z_i^{(n_l)}))^2 \\ &=-(y_i-a_i^{(n_l)})g'(z_i^{(n_l)})\\ \delta_i^{(l)}&=\frac{\partial J(\theta)}{\partial z_i^{(l)}}\\ &=\sum_{j=1}^{S_{l+1}}\frac{\partial J(\theta)}{\partial z_j^{(l+1)}}\frac{\partial z_j^{(l+1)}}{\partial z_i^{(l)}}\\ &=\sum_{j=1}^{S_{l+1}}\delta_j^{(l+1)}\frac{\partial z_j^{(l+1)}}{\partial z_i^{(l)}}\\ &=\sum_{j=1}^{S_{l+1}}\delta_j^{(l+1)}\frac{\partial}{\partial z_i^{(l)}}(b_j^{(l)}+\sum_{k=1}^{S_l}w_{jk}^{(l)}a_k^{(l)}) \\ &=\sum_{j=1}^{S_{l+1}}\delta_j^{(l+1)}\frac{\partial}{\partial z_i^{(l)}}(b_j^{(l)}+\sum_{k=1}^{S_l}w_{jk}^{(l)}g(z_k^{(l)})) \\ &=\sum_{j=1}^{S_{l+1}}\delta_j^{(l+1)}\frac{\partial}{\partial z_i^{(l)}}(w_{ji}^{(l)}g(z_i^{(l)})) \\ &=\sum_{j=1}^{S_{l+1}}\delta_j^{(l+1)}w_{ji}^{(l)}g'(z_i^{(l)}) \\ &=g'(z_i^{(l)})\sum_{j=1}^{S_{l+1}}\delta_j^{(l+1)}w_{ji}^{(l)} \\ \frac{\partial J(\theta)}{\partial w_{ij}^{(l)}}&=\frac{\partial J(\theta)}{\partial z_i^{(l+1)}}\frac{\partial z_i^{(l+1)}}{\partial w_{ij}^{(l)}}\\ &=\delta _i^{(l+1)}\frac{\partial z_i^{(l+1)}}{\partial w_{ij}^{(l)}}\\ &=\delta _i^{(l+1)}\frac{\partial}{\partial w_{ij}^{(l)}}(b_i^{(l)}+\sum_{k=1}^{S_l}w_{ik}^{(l)}a_k^{(l)}) \\ &=\delta _i^{(l+1)}a_j^{(l)}\\ \frac{\partial J(\theta)}{\partial b_i^{(l)}}&=\delta _i^{(l+1)}\frac{\partial}{\partial b_i^{(l)}}(b_i^{(l)}+\sum_{k=1}^{S_l}w_{ik}^{(l)}a_k^{(l)}) \\ &=\delta _i^{(l+1)} \end{align*}$

向量形式的公式

\begin{aligned} δ^{(l)} & = (W^{(l)})^{T} δ^{(l + 1)} \circ g^{'} (z^{(l)}) \\ \frac{\partial J (θ)}{\partial W^{(l)}} & = δ^{(l + 1)} (a^{(l)})^{T} \\ \frac{\partial J (θ)}{\partial b^{(l)}} & = δ^{(l + 1)} \end{aligned}

$\begin{align*} \boldsymbol{\delta}^{(l)}&=(\boldsymbol{W}^{(l)})^T\boldsymbol{\delta}^{(l+1)}\circ g'(\boldsymbol{z}^{(l)})\\ \frac{\partial J(\theta)}{\partial \boldsymbol{W}^{(l)}}&=\boldsymbol{\delta}^{(l+1)}(\boldsymbol{a}^{(l)})^T\\ \frac{\partial J(\theta)}{\partial \boldsymbol{b}^{(l)}}&=\boldsymbol{\delta}^{(l+1)} \end{align*}$
其中，

\circ

$\circ$ 表示每个元素相乘，粗体的小写符号表示列向量，粗体的大写符号表示矩阵。

参考

（[1] 中的公式推导有错误，本文已纠正）
[1] https://www.cnblogs.com/nowgood/p/backprop.html
[2] Bouvrie J. Notes on convolutional neural networks[J]. 2006.

反向传播算法的公式推导

概念