backpropagation算法示例
下面举个例子，假设在某个mini-batch的有样本X和标签Y，其中\(X\in R^{m\times 2}, Y\in R^{m\times 1}\)，现在有个两层的网络，对应的计算如下：
\[ \begin{split} i_1 &= XW_1+ b_1\\ o_1 &= sigmoid(i_1)\\ i_2 &= o_1W_2 + b_2\\ o_2 &= sigmoid(i_2) \end{split} \]
其中\(W_1 \in R^{2\times 3}, b_1\in R^{1\times 3}, W_2\in R^{3\times 1}, b_2\in R^{1\times 1}\)都是参数，然后使用平方损失函数
\[ cost = \dfrac{1}{2m}\sum_i^m(o_{2i} - Y_i)^2 \]
下面给出反向传播的过程
\[ \begin{split} \dfrac{\partial cost}{\partial o_2} &= \dfrac{1}{m}(o_2 - Y)\\ \dfrac{\partial o_2}{\partial i_2} &= sigmoid(i_2)*(1 - sigmoid(i_2))\\ \dfrac{\partial i_2}{\partial W_2} &= o_1\\ \dfrac{\partial i_2}{\partial b_2} &= 1\\ \dfrac{\partial o_1}{\partial i_1} &= sigmoid(i_1)*(1 - sigmoid(i_1))\\ \dfrac{\partial i_1}{\partial W_1} &= X\\ \dfrac{\partial i_1}{\partial b_1} &= 1 \end{split} \]
所以有
\[ \begin{split} \Delta W_2 &= (\dfrac{\partial i_2}{\partial W_2})^T\times (\dfrac{\partial cost}{\partial o_2}*\dfrac{\partial o_2}{\partial i_2})\\ \Delta b_2 &= (\dfrac{\partial i_2}{\partial b_2})^T\times (\dfrac{\partial cost}{\partial o_2}*\dfrac{\partial o_2}{\partial i_2}) \end{split} \]