0_2_1-卷积层的反向传播-单通道、无padding、步长1

卷积网络包含一系列的卷积层，每层由输入特征图 $I$ ，一堆滤波器 $K$ 和偏置 $b$ . 假设输入的高度、宽度、通道数分别为 $H,W,C$ ; 则 $I \in \Bbb R^{H \times W \times C}$ , 输出 $D$ 个通道的卷积层，则有卷积核 $K \in \Bbb R^{k_1 \times k_2 \times C \times D}$ ,偏置 $b \in \Bbb R^D$ ,每个输出通道一个偏置。则其中一个输出通道的可以如下表示：

\begin{aligned} (2) & (I * K)_{i j} & = \sum_{m = 0}^{k_{1} - 1} \sum_{n = 0}^{k_{2} - 1} \sum_{c = 1}^{C} K_{m, n, c} \cdot I_{i + m, j + n, c} + b \end{aligned}

$\begin{align} (I \ast K)_{ij} &= \sum_{m = 0}^{k_1 - 1} \sum_{n = 0}^{k_2 - 1} \sum_{c = 1}^{C} K_{m,n,c} \cdot I_{i+m, j+n, c} + b \tag {2} \end{align}$
有心读者肯定会疑惑公式(2),没有体现padding和卷积核的步长。由于卷积相对复杂，没有办法一次性说明的非常清楚，计划分几次来逐步说明；本文接下来将推导最简单的卷积反向传播公式。假定输入输出通道都为1，即

C = D = 1

$C=D=1$ , 且卷积核的padding=0,步长为1。

约定说明

a) $l$ 代表网络的第 $l$ 层, $z^l$ 代表第 $l$ 层卷积， $z_{i,j}^l$ 代表第 $l$ 层卷积的 $(i,j)$ 位置的值; $z^l$ 的高度和宽度分别为 $H^l,\hat W^l$ ( $\color{red}{避免与权重相同}$ )

b) $W^{l-1},b^{l-1}$ 代表连接第 $l-1$ 层和第 $l$ 层的卷积核权重和偏置; 卷积核的维度为 $(k_1^{l-1},k_2^{l-1})$ 。

c) 记损失函数L关于第 $l$ 层卷积的输出 $z^l$ 的偏导为 $\delta^l = \frac {\partial L} {\partial z^l} \ \ \ (3)$

根据以上约定第 $l$ 层卷积输出为:

\begin{aligned} (4) & z_{i, j}^{l} = \sum_{m = 0}^{k_{1}^{l - 1} - 1} \sum_{n = 0}^{k_{2}^{l - 1} - 1} W_{m, n}^{l - 1} z_{i + m, j + n}^{l - 1} + b^{l - 1} & i \in [0, H^{l} - 1], j \in [0, {\hat{W}}^{l} - 1] \end{aligned}

$\begin{align} &z^l_{i,j} = \sum_{m=0}^{k_1^{l-1}-1} \sum_{n=0}^{k_2^{l-1}-1} W_{m,n}^{l-1} z_{i+m,j+n}^{l-1} + b^{l-1} & i \in [0,H^l-1], j\in [0,\hat W^l-1]\tag 4 \end{align}$
其中：

H^{l} = H^{l - 1} - k_{1}^{l - 1} + 1; {\hat{W}}^{l} = {\hat{W}}^{l - 1} - k_{2}^{l - 1} + 1

$H^l = H^{l-1} - k_1^{l-1} + 1;\ \ \ \ \ \hat W^l = \hat W^{l-1} - k_2^{l-1} + 1$

误差反向传播

权重梯度

a) 首先来看损失函数 $L$ 关于第 $l-1$ 层权重 $W^{l-1}$ 和偏置 $b^{l-1}$ 的梯度

\begin{aligned} (1) & \frac{\partial L}{\partial W_{m, n}^{l - 1}} = \sum_{i} \sum_{j} \frac{\partial L}{\partial z_{i, j}^{l}} * \frac{\partial z^{l} i, j}{\partial W_{m, n}^{l - 1}} & / / l 层 的 每 个 神 经 元 都 有 梯 度 传 给 权 重 W_{m, n}^{l - 1} \\ (2) & = \sum_{i} \sum_{j} δ_{i, j}^{l} * \frac{\partial (\sum_{m = 0}^{k_{1}^{l - 1}} \sum_{n = 0}^{k_{2}^{l - 1}} W_{m, n}^{l - 1} z_{i + m, j + n}^{l - 1} + b^{l - 1})}{\partial W_{m, n}^{l - 1}} \\ (5) & = \sum_{i} \sum_{j} δ_{i, j}^{l} * z_{i + m, j + n}^{l - 1} \end{aligned}

$\begin{align} &\frac {\partial L} {\partial W_{m,n}^{l-1}} = \sum_i \sum_j \frac {\partial L} {\partial z^l_{i,j}} * \frac {\partial z^l{i,j}} {\partial W_{m,n}^{l-1}} &//l层的每个神经元都有梯度传给权重W^{l-1}_{m,n}\\ &=\sum_i \sum_j \delta^l_{i,j} * \frac {\partial ( \sum_{m=0}^{k_1^{l-1}} \sum_{n=0}^{k_2^{l-1}} W_{m,n}^{l-1} z_{i+m,j+n}^{l-1} + b^{l-1})} {\partial W^{l-1}_{m,n}} \\ &=\sum_i \sum_j \delta^l_{i,j} * z^{l-1}_{i+m,j+n} \tag 5 \end{align} \\$

对比公式(5)和公式(4),可以发现,损失函数 $L$ 关于第 $l-1$ 层权重 $W^{l-1}$ 的梯度就是以 $\delta^l$ 为卷积核在 $z^{l-1}$ 上做卷积的结果(这里没有偏置项)。多么简介对称呀!!!。

b) 同理可得

\begin{aligned} (6) & \frac{\partial L}{\partial b^{l - 1}} = \sum_{i} \sum_{j} δ_{i, j}^{l} \end{aligned}

$\begin{align} \frac {\partial L} {\partial b^{l-1}} =\sum_i \sum_j \delta^l_{i,j} \tag 6 \end{align}$

l-1层梯度

然后再来看看损失函数关于第 $l-1$ 层输出的梯度

\begin{aligned} (3) & δ_{i^{'}, j^{'}}^{l - 1} = \frac{\partial L}{\partial z_{i^{'}, j^{'}}^{l - 1}} = \sum_{i} \sum_{j} \frac{\partial L}{\partial z_{i, j}^{l}} * \frac{\partial z_{i, j}^{l}}{\partial z_{i^{'}, j^{'}}^{l - 1}} \\ (4) & = \sum_{i} \sum_{j} δ_{i, j}^{l} * \frac{\partial (\sum_{m = 0}^{k_{1}^{l - 1}} \sum_{n = 0}^{k_{2}^{l - 1}} W_{m, n}^{l - 1} z_{i + m, j + n}^{l - 1} + b^{l - 1})}{\partial z_{i^{'}, j^{'}}^{l - 1}} & / / 当 i = i^{'} - m, j = j^{'} - n 时 有 梯 度 W_{m, n}^{l - 1} \\ (5) & = \sum_{i} \sum_{j} δ_{i, j}^{l} W_{m, n}^{l - 1} & / / 此 时 m = i^{'} - i, n = j^{'} - j \\ (6) & = \sum_{m} \sum_{n} δ_{i^{'} - m, j^{'} - n}^{l} W_{m, n}^{l - 1} (7) & / / 此 时 i = i^{'} - m \in [0, H^{l} - 1], j = j^{'} - n \in [0, {\hat{W}}^{l} - 1] \\ (7) & = \sum_{i} \sum_{j} δ_{i, j}^{l} W_{i^{'} - i, j^{'} - j}^{l - 1} (8) & / / 需 要 满 足 i^{'} - i \in [0, k_{1}^{l - 1} - 1], j^{'} - j \in [0, k_{2}^{l - 1} - 1] \end{aligned}

$\begin{align} &\delta^{l-1}_{i^{\prime},j^{\prime}}=\frac {\partial L} {\partial z_{i^{\prime},j^{\prime}}^{l-1}} = \sum_i \sum_j \frac {\partial L} {\partial z^l_{i,j}} * \frac {\partial z^l_{i,j}} {\partial z_{i^{\prime},j^{\prime}}^{l-1}} \\ &=\sum_i \sum_j \delta^l_{i,j} * \frac {\partial ( \sum_{m=0}^{k_1^{l-1}} \sum_{n=0}^{k_2^{l-1}} W_{m,n}^{l-1} z_{i+m,j+n}^{l-1} + b^{l-1})} {\partial z_{i^{\prime},j^{\prime}}^{l-1}} &//当i=i^{\prime}-m, j=j^{\prime}-n时有梯度W^{l-1}_{m,n}\\ &=\sum_i \sum_j \delta^l_{i,j} W^{l-1}_{m,n} &//此时m=i^{\prime}-i ,n=j^{\prime}-j\\ &=\sum_m \sum_n \delta^l_{i^{\prime}-m,j^{\prime}-n}W^{l-1}_{m,n} \ \ \ \ \ \ \ (7) &//此时i=i^{\prime}-m \in[0,H^l-1],j=j^{\prime}-n \in [0,\hat W^l-1] \\ &=\sum_i \sum_j \delta^l_{i,j} W^{l-1}_{i^{\prime}-i,j^{\prime}-j} \ \ \ \ \ \ \ (8) &//需要满足i^{\prime}-i \in [0,k_1^{l-1}-1],j^{\prime}-j \in [0,k_2^{l-1}-1] \end{align}$

约束条件: $i^{\prime}-i \in [0,k_1^{l-1}-1],j^{\prime}-j \in [0,k_2^{l-1}-1]$

变换一下就是: $i \in [i^{\prime}+1-k_1^{l-1},i^{\prime}],j \in [j^{\prime}+1-k_2^{l-1},j^{\prime}] \tag 9$

同时 $i,j$ 需要满足公式(4)的约束条件:

\begin{matrix} (10) & i \in [0, H^{l} - 1], j \in [0, {\hat{W}}^{l} - 1] \end{matrix}

$i\in [0,H^l-1], j\in [0,\hat W^l-1] \tag {10}$
因此有

\begin{matrix} (11) & {\begin{cases} i \in [max (0, i^{'} + 1 - k_{1}^{l - 1}), min (H^{l} - 1, i^{'})] \\ j \in [max (0, j^{'} + 1 - k_{2}^{l - 1}), min ({\hat{W}}^{l} - 1, j^{'})] \end{cases} \end{matrix}

$\begin{cases} i \in [\max(0,i^{\prime}+1-k_1^{l-1}),\min(H^l-1,i^{\prime})] \\ j \in [\max(0,j^{\prime}+1-k_2^{l-1}),\min(\hat W^l-1,j^{\prime})] \tag {11} \end{cases}$
下面来看一个例子，对于l-1层

5 \times 5

$5 \times 5$ 的卷积层，卷积核

3 \times 3

$3 \times 3$ , 则输出的l层卷积大小为5-3-1=3，也就是

3 \times 3

$3 \times 3$ , 此时有：

{\begin{cases} i \in [max (0, i^{'} - 2), min (2, i^{'})] \\ j \in [max (0, j^{'} - 2, min (2, j^{'})] \end{cases}

$\begin{cases} i \in [\max(0,i^{\prime}-2),\min(2,i^{\prime})] \\ j \in [\max(0,j^{\prime}-2,\min(2,j^{\prime})] \end{cases}$
根据公式(7)及其约束条件有：

\begin{aligned} (8) & δ_{0, 0}^{l - 1} = δ_{0, 0}^{l} W_{0, 0}^{l - 1} & i \in [0, 0], j \in [0, 0] \\ (9) & δ_{0, 1}^{l - 1} = δ_{0, 1}^{l} W_{0, 0}^{l - 1} + δ_{0, 0}^{l} W_{0, 1}^{l - 1} & i \in [0, 0], j \in [0, 1] \\ (10) & δ_{0, 2}^{l - 1} = δ_{0, 2}^{l} W_{0, 0}^{l - 1} + δ_{0, 1}^{l} W_{0, 1}^{l - 1} + δ_{0, 0}^{l} W_{0, 2}^{l - 1} & i \in [0, 0], j \in [0, 2] \\ (11) & δ_{1, 0}^{l - 1} = δ_{1, 0}^{l} W_{0, 0}^{l - 1} + δ_{0, 0}^{l} W_{1, 0}^{l - 1} & i \in [0, 1], j \in [0, 0] \\ (12) & δ_{1, 1}^{l - 1} = δ_{1, 1}^{l} W_{0, 0}^{l - 1} + δ_{0, 1}^{l} W_{1, 0}^{l - 1} + δ_{1, 0}^{l} W_{0, 1}^{l - 1} + δ_{0, 0}^{l} W_{1, 1}^{l - 1} & i \in [0, 1], j \in [0, 1] \\ (13) & δ_{1, 2}^{l - 1} = \sum_{i} \sum_{j} δ_{i, j}^{l} W_{i^{'} - i, j^{'} - j}^{l - 1} & i \in [0, 1], j \in [0, 2] \\ (14) & . . . . . . \\ (15) & δ_{2, 2}^{l - 1} = \sum_{i} \sum_{j} δ_{i, j}^{l} W_{i^{'} - i, j^{'} - j}^{l - 1} & i \in [0, 2], j \in [0, 2] \end{aligned}

$\begin{align} &\delta^{l-1}_{0,0} =\delta^{l}_{0,0}W^{l-1}_{0,0} &i \in [0,0],j \in [0,0] \\ &\delta^{l-1}_{0,1} =\delta^{l}_{0,1}W^{l-1}_{0,0} + \delta^{l}_{0,0}W^{l-1}_{0,1} &i \in [0,0],j \in [0,1] \\ &\delta^{l-1}_{0,2} =\delta^{l}_{0,2}W^{l-1}_{0,0} + \delta^{l}_{0,1}W^{l-1}_{0,1} +\delta^{l}_{0,0}W^{l-1}_{0,2} &i \in [0,0],j \in [0,2] \\ &\delta^{l-1}_{1,0} =\delta^{l}_{1,0}W^{l-1}_{0,0} + \delta^{l}_{0,0}W^{l-1}_{1,0} &i \in [0,1],j \in [0,0] \\ &\delta^{l-1}_{1,1} =\delta^{l}_{1,1}W^{l-1}_{0,0} + \delta^{l}_{0,1}W^{l-1}_{1,0} +\delta^{l}_{1,0}W^{l-1}_{0,1} + \delta^{l}_{0,0}W^{l-1}_{1,1} &i \in [0,1],j \in [0,1] \\ &\delta^{l-1}_{1,2} = \sum_i \sum_j \delta^l_{i,j} W^{l-1}_{i^{\prime}-i,j^{\prime}-j} & i \in [0,1],j \in [0,2] \\ &... ... \\ &\delta^{l-1}_{2,2} = \sum_i \sum_j \delta^l_{i,j} W^{l-1}_{i^{\prime}-i,j^{\prime}-j} & i \in [0,2],j \in [0,2] \\ \end{align}$

等价于以下的卷积

δ^{l - 1} = (\begin{aligned} (16) & 0, & 0, & 0, & 0, & 0, & 0, & 0 \\ (17) & 0, & 0, & 0, & 0, & 0, & 0, & 0 \\ (18) & 0, & 0, & δ_{0, 0}^{l}, & δ_{0, 1}^{l}, & δ_{0, 2}^{l}, & 0, & 0 \\ (19) & 0, & 0, & δ_{1, 0}^{l}, & δ_{1, 1}^{l}, & δ_{1, 2}^{l}, & 0, & 0 \\ (20) & 0, & 0, & δ_{2, 0}^{l}, & δ_{2, 1}^{l}, & δ_{2, 2}^{l}, & 0, & 0 \\ (21) & 0, & 0, & 0, & 0, & 0, & 0, & 0 \\ (22) & 0, & 0, & 0, & 0, & 0, & 0, & 0 \end{aligned}) * (\begin{matrix} W_{2, 2}^{l - 1}, & W_{2, 1}^{l - 1}, & W_{2, 0}^{l - 1} \\ W_{1, 2}^{l - 1}, & W_{11}^{l - 1}, & W_{1, 0}^{l - 1} \\ W_{0, 2}^{l - 1}, & W_{01}^{l - 1}, & W_{0, 0}^{l - 1} \end{matrix})

$\delta^{l-1}=\left( \begin{align} &0, &&0,&&0,&&0,&&0,&&0,&&0 \\ &0, &&0,&&0,&&0,&&0,&&0,&&0 \\ &0,&&0,&&\delta^{l}_{0,0},&&\delta^{l}_{0,1},&&\delta^{l}_{0,2},&&0,&&0\\ &0,&&0,&&\delta^{l}_{1,0},&&\delta^{l}_{1,1},&&\delta^{l}_{1,2},&&0,&&0\\ &0,&&0,&&\delta^{l}_{2,0},&&\delta^{l}_{2,1},&&\delta^{l}_{2,2},&&0,&&0\\ &0,&&0, &&0,&&0,&&0,&&0,&&0 \\ &0,&&0, &&0,&&0,&&0,&&0,&&0 \end{align} \right) * \left( \begin{array} aW^{l-1}_{2,2},& W^{l-1}_{2,1},& W^{l-1}_{2,0}\\ W^{l-1}_{1,2},& W^{l-1}_{11},& W^{l-1}_{1,0}\\ W^{l-1}_{0,2},& W^{l-1}_{01},& W^{l-1}_{0,0}\\ \end{array} \right)$
即以

W^{l - 1}

$W^{l-1}$ 翻转

180^{\circ}

$180^\circ$ 的矩阵为卷积核在

δ^{l}

$\delta^l$ 加上padding=2的矩阵上做卷积的结果。

a) 设 $rot_{180^\circ}W^{l-1}$ 为以 $W^{l-1}$ 翻转 $180^\circ$ 的矩阵后的矩阵

b) 设 $p\delta^l$ 为 $\delta^l$ 加上padding高宽为卷积核高宽减1即 $(k_1^{l-1}-1,k_2^{l-1}-1)$ 后的梯度矩阵，可知其高度为 $H^l+2k_1^{l-1} -2 = H^{l-1} +k_1^{l-1} -1$ ;相应的宽度为 $\hat W^{l-1} +k_2^{l-1} -1$

c) 卷积核 $rot_{180^\circ}W^{l-1}$ 的大小为 $(k_1^{l-1},k_1^{l-1})$ ,在上做完卷积后的长宽刚好与 $\delta^{l-1}$ 的高度和宽度一样，即 $(H^{l-1},\hat W^{l-1})$ 。

d) $p\delta^l$ 和 $\delta^l$ 的关系如下：

\begin{matrix} (12) & p δ_{i, j}^{l} = {\begin{cases} δ_{i - k_{1}^{l - 1} + 1, j - k_{2}^{l - 1} + 1}^{l} & i \in [k_{1}^{l - 1} - 1, H^{l} + k_{1}^{l - 1} - 2] 且 j \in [k_{2}^{l - 1} - 1, {\hat{W}}^{l} + k_{2}^{l - 1} - 2] \\ 0 & i, j 其 它 情 况 \end{cases} \end{matrix}

$p\delta^l_{i,j}=\begin{cases} \delta^l_{i-k_1^{l-1}+1,j-k_2^{l-1}+1} & i \in[k_1^{l-1}-1,H^l+k_1^{l-1}-2] 且j \in [k_2^{l-1}-1,\hat W^l+k_2^{l-1}-2] \\ 0 & i,j其它情况 \tag {12} \end{cases}$

接下来将证明这个卷积就是 $\delta^{l-1}$

根据公式(4) 卷积后的 $(i,j)$ 位置的值为：

\begin{aligned} (23) & \sum_{m = 0}^{k_{1}^{l - 1} - 1} \sum_{n = 0}^{k_{2}^{l - 1} - 1} r o t_{180^{\circ}} W_{m, n}^{l - 1} p δ_{i + m, j + n}^{l} \\ (24) & = \sum_{m = 0}^{k_{1}^{l - 1} - 1} \sum_{n = 0}^{k_{2}^{l - 1} - 1} W_{k_{1}^{l - 1} - 1 - m, k_{2}^{l - 1} - 1 - n} p δ_{i + m, j + n}^{l} / / 将 翻 转 180^{\circ} 改 回 来 \\ (25) & = \sum_{m^{'} = 0}^{k_{1}^{l - 1} - 1} \sum_{n^{'} = 0}^{k_{2}^{l - 1} - 1} W_{m^{'}, n^{'}} p δ_{i + k_{1}^{l - 1} - 1 - m^{'}, j + k_{2}^{l - 1} - 1 - n^{'}}^{l} / / m^{'} + m = k_{1}^{l - 1} - 1, n^{'} + n = k_{2}^{l - 1} - 1 \\ (26) & = \sum_{m = 0}^{k_{1}^{l - 1} - 1} \sum_{n = 0}^{k_{2}^{l - 1} - 1} W_{m, n} p δ_{i + k_{1}^{l - 1} - 1 - m, j + k_{2}^{l - 1} - 1 - n}^{l} / / 将 下 标 改 回 来 \\ (15) & = \sum_{m = 0}^{k_{1}^{l - 1} - 1} \sum_{n = 0}^{k_{2}^{l - 1} - 1} W_{m, n} {\begin{cases} δ_{i - m, j - n}^{l} & / / i - m \in [0, H^{l} - 1] 且 j - n \in [0, {\hat{W}}^{l} - 1] \\ 0 & / / i - m \notin [0, H^{l} - 1] 或 j - n \notin [0, {\hat{W}}^{l} - 1] \end{cases} \end{aligned}

$\begin{align} &\sum_{m=0}^{k_1^{l-1}-1} \sum_{n=0}^{k_2^{l-1}-1}rot_{180^\circ} W^{l-1}_{m,n}p\delta^{l}_{i+m,j+n} \\ &=\sum_{m=0}^{k_1^{l-1}-1} \sum_{n=0}^{k_2^{l-1}-1}W_{k^{l-1}_1-1-m,k_2^{l-1}-1-n}\ p\delta^{l}_{i+m,j+n} \ \ \ \ \ \ //将翻转180^\circ改回来 \\ &=\sum_{m^{\prime}=0}^{k_1^{l-1}-1} \sum_{n^{\prime}=0}^{k_2^{l-1}-1}W_{m^{\prime},n^{\prime}}\ p\delta^{l}_{i+k^{l-1}_1-1-m^{\prime},j+k_2^{l-1}-1-n^{\prime}} \ \ \ \ \ //m^{\prime} +m =k_1^{l-1} -1,n^{\prime}+n=k_2^{l-1}-1 \\ &=\sum_{m=0}^{k_1^{l-1}-1} \sum_{n=0}^{k_2^{l-1}-1}W_{m,n}\ p\delta^{l}_{i+k^{l-1}_1-1-m,j+k_2^{l-1}-1-n} \ \ \ \ \ //将下标改回来 \\ &= \sum_{m=0}^{k_1^{l-1}-1} \sum_{n=0}^{k_2^{l-1}-1}W_{m,n}\ \begin{cases} \delta^{l}_{i-m,j-n} &//i-m \in [0,H^l-1] 且j-n \in[0,\hat W^l-1] \\ 0 &//i-m \notin [0,H^l-1] 或j-n \notin[0,\hat W^l-1] \tag {15} \end{cases} \end{align}$
可以看出公式(15)与公式(7)完全一致。

结论

a) 卷积前向计算公式如下:

\begin{aligned} (4) & z_{i, j}^{l} = \sum_{m = 0}^{k_{1}^{l - 1} - 1} \sum_{n = 0}^{k_{2}^{l - 1} - 1} W_{m, n}^{l - 1} z_{i + m, j + n}^{l - 1} + b^{l - 1} & i \in [0, H^{l} - 1], j \in [0, {\hat{W}}^{l} - 1] \end{aligned}

$\begin{align} &z^l_{i,j} = \sum_{m=0}^{k_1^{l-1}-1} \sum_{n=0}^{k_2^{l-1}-1} W_{m,n}^{l-1} z_{i+m,j+n}^{l-1} + b^{l-1} & i \in [0,H^l-1], j\in [0,\hat W^l-1]\tag 4 \end{align}$
b) 损失函数

L

$L$ 关于第

l - 1

$l-1$ 层权重

W^{l - 1}

$W^{l-1}$ 的梯度，是以损失函数

L

$L$ 关于第

l

$l$ 层梯度

δ^{l}

$\delta^l$ 为卷积核在

z^{l - 1}

$z^{l-1}$ 上做卷积的结果

\begin{matrix} (5) & \frac{\partial L}{\partial W_{m, n}^{l - 1}} = \sum_{i} \sum_{j} δ_{i, j}^{l} * z_{i + m, j + n}^{l - 1} \end{matrix}

$\frac {\partial L} {\partial W_{m,n}^{l-1}} =\sum_i \sum_j \delta^l_{i,j} * z^{l-1}_{i+m,j+n} \tag 5$
c) 损失函数

L

$L$ 关于第

l - 1

$l-1$ 层偏置

b^{l - 1}

$b^{l-1}$ 的梯度，是

δ^{l}

$\delta^l$ 元素之和

\begin{matrix} (6) & \frac{\partial L}{\partial b^{l - 1}} = \sum_{i} \sum_{j} δ_{i, j}^{l} \end{matrix}

$\frac {\partial L} {\partial b^{l-1}} =\sum_i \sum_j \delta^l_{i,j} \tag 6$

d) 以损失函数 $L$ 关于第 $l-1$ 层梯度 $\delta^{l-1}$ , 是以第 $l-1$ 层权重的翻转 $rot_{180^\circ} W^{l-1}$ 为卷积核在 $\delta^l$ 加上padding高宽为 $(k_1^{l-1}-1,k_2^{l-1}-1)$ 后的梯度矩阵 $p\delta^{l}$ 上卷积

\begin{matrix} (16) & δ^{l - 1} = \sum_{m = 0}^{k_{1}^{l - 1} - 1} \sum_{n = 0}^{k_{2}^{l - 1} - 1} r o t_{180^{\circ}} W_{m, n}^{l - 1} p δ_{i + m, j + n}^{l} \end{matrix}

$\delta^{l-1}=\sum_{m=0}^{k_1^{l-1}-1} \sum_{n=0}^{k_2^{l-1}-1}rot_{180^\circ} W^{l-1}_{m,n}p\delta^{l}_{i+m,j+n} \tag {16}$