深度学习：神经网络从1到N

符号表示

• Superscript $[l]$ denotes a quantity associated with the $l^{th}$ layer.
  • Example: $a^{[L]}$ is the $L^{th}$ layer activation. $W^{[L]}$ and $b^{[L]}$ are the $L^{th}$ layer parameters.
• Superscript $(i)$ denotes a quantity associated with the $i^{th}$ example.
  • Example: $x^{(i)}$ is the $i^{th}$ training example.
• Lowerscript $i$ denotes the $i^{th}$ entry of a vector.
  • Example: $a^{[l]}_i$ denotes the $i^{th}$ entry of the $l^{th}$ layer’s activations).
• $\frac{\partial J}{\partial a } = da$ for any variable $a$.

1. 一层神经网络

逻辑回归实际就是一层神经网络

$x=(x_1, x_2)^T,w=(w_1, w_2)^T$ $z = w^T x + b = w_1x_1+w_2x_2+b$则 $\frac{\partial z}{\partial w}= x = (x_1, x_2)^T$

$\frac{\partial z}{\partial b} = 1$

$a = \sigma(z) = \frac{1}{1+e^{-z}}$，则 $\frac{\partial a}{\partial z} = a(1-a)$

$L(a,y)=-[y\log a+(1-y)\log(1-a)]$，则 $\frac{\partial L}{\partial a} = - \frac{y}{a}+\frac{1-y}{1-a} = \frac{a-y}{a(1-a)}$根据链式法则：
$\begin{aligned} \frac{\partial L}{\partial w} =\frac{\partial L}{\partial a} \frac{\partial a}{\partial z} \frac{\partial z}{\partial w} &= (a-y)x \end{aligned}$

$\begin{aligned} \frac{\partial L}{\partial b} =\frac{\partial L}{\partial a} \frac{\partial a}{\partial z} \frac{\partial z}{\partial b} &= (a-y)\end{aligned}$

For $i^{th}$ training example $x^{(i)}(i=1,...,m)$:
$z^{(i)} = w^T x^{(i)} + b$

$\hat {y}^{(i)} = a^{(i)} = \sigma(z^{(i)})$

$\mathcal{L}(a^{(i)}, y^{(i)}) = - [y^{(i)} \log a^{(i)} + (1-y^{(i)} ) \log(1-a^{(i)})]$

For $X = (x^{(0)}, x^{(1)}, ..., x^{(m)})$
$Z=w^T X +b, \;\;\; Z= (z^{(0)}, z^{(1)}, ...,z^{(m)})$

$\hat{Y}= A = \sigma(Z), \;\;\; A= (a^{(0)}, a^{(1)}, ..., a^{(m)})$

$J = \frac{1}{m} \sum_{i=1}^m \mathcal{L}(a^{(i)}, y^{(i)})=-\frac{1}{m}[\sum_{i=1}^{m}y^{(i)}\log a^{(i)}+(1-y^{(i)})\log(1-a^{(i)})]$则 $\frac{\partial J}{\partial w} = \frac{1}{m}X(A-Y)^T$ $\frac{\partial J}{\partial b} = \frac{1}{m} \sum_{i=1}^m (a^{(i)}-y^{(i)})$

2. 两层神经网络

（1）Forward Propagation

Input Layer

$x=(x_1,x_2)^T$

Hidden Layer

$z_1^{[1]} = {w_1^{[1]}}^Tx + b_1^{[1]} \;\;\;\; a_1^{[1]}= g(z_1^{[1]})$

$z_2^{[1]} = {w_2^{[1]}}^Tx + b_2^{[1]} \;\;\;\; a_2^{[1]}= g(z_2^{[1]})$

$z_3^{[1]} = {w_3^{[1]}}^Tx + b_3^{[1]} \;\;\;\; a_3^{[1]}= g(z_3^{[1]})$

$z_4^{[1]} = {w_4^{[1]}}^Tx + b_4^{[1]} \;\;\;\; a_4^{[1]}= g(z_4^{[1]})$

令 $z^{[1]}=(z_1^{[1]}, z_2^{[1]}, z_3^{[1]}, z_4^{[1]})^T,\;\; W^{[1]}=({w_1^{[1]}}^T,{w_2^{[1]}}^T,{w_3^{[1]}}^T,{w_4^{[1]}}^T)^T, \;\; b^{[1]}=( b_1^{[1]}, b_2^{[1]}, b_3^{[1]}, b_4^{[1]})^T$
则 $z^{[1]} = {W^{[1]}}x + b^{[1]} \;\;\;\; a^{[1]}= g(z^{[1]})$

Output Layer

$z^{[2]} = {W^{[2]}}a^{[1]} + b^{[2]} \;\;\;\; a^{[2]}=\sigma(z^{[2]})$

For $x^{(i)}(i=1,...,m)$:
$z^{[1] (i)} = W^{[1]} x^{(i)} + b^{[1]} \;\;\;\; a^{[1] (i)} = g(z^{[1] (i)})$

$z^{[2] (i)} = W^{[2]} a^{[1] (i)} + b^{[2]} \;\;\;\;\; \hat{y}^{(i)} = a^{[2] (i)} = \sigma(z^{ [2] (i)})$

$y^{(i)}_{prediction} = \begin{cases} 1 {if } \;\;\;\;\;\hat{y}^{(i)} > 0.5 \\ 0 \;\;\;\;\;\;\;\;{otherwise } \end{cases}$

For $X = (x^{(0)}, x^{(1)}, ..., x^{(m)})$:
$Z^{[1]} = W^{[1]}X + b^{[1]}\;\;\;\;A^{[1]} = g(Z^{[1]})$

$Z^{[2]} = W^{[2]} A^{[1]} + b^{[2]}\;\;\;\;\; \hat{Y} = A^{[2]} = \sigma(Z^{[2]})$

其中， $Z^{[l]}=(z^{[l] (1)},z^{[l] (2)},...,z^{[l] (m)})\;\;\;A^{[l]}=(a^{[l] (1)},a^{[l] (2)},...,a^{[l] (m)})$

$J = - \frac{1}{m} \sum\limits_{i = 0}^{m} \large\left(\small y^{(i)}\log\left(a^{[2] (i)}\right) + (1-y^{(i)})\log\left(1- a^{[2] (i)}\right) \large \right) \small$

（2） Borward Propagation

3. 三层神经网络

3_Layers_NN ： LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID

$m=5,L=3,n^{[0]}=n^{[x]}=3, n^{[1]}=2, n^{[2]}=3, n^{[3]}=1$

（1） Forward Propatation

$J = - \frac{1}{m} \sum\limits_{i = 0}^{m} \large\left(\small y^{(i)}\log\left(a^{[3] (i)}\right) + (1-y^{(i)})\log\left(1- a^{[3] (i)}\right) \large \right) \small$

（2） Borward Propatation

Layer 3：
$dZ^{[3]} = A^{[3]} - Y$ $dW^{[3]} = \frac{1}{m}dZ^{[3]} A^{[2]^{T}}$ $db^{[3]} = \frac{1}{m} \sum_{i = 1}^{m} dz^{[3](i)}= \frac{1}{m} np.sum(dZ^{[3]},axis=1, keepdim=True)$
Layer 2:
$dA^{[2]} = W^{[3]^{T}} dZ^{[3]}$ $dZ^{[2]} = dA^{[2]}*g'(Z^{[2]})$ $dW^{[2]} = \frac{1}{m}dZ^{[2]} A^{[1]^{T}}$ $db^{[2]} = \frac{1}{m} \sum_{i = 1}^{m} dz^{[2](i)}= \frac{1}{m} np.sum(dZ^{[2]},axis=1, keepdim=True)$
Layer 1:
$dA^{[1]} = W^{[2]^{T}} dZ^{[2]}$ $dZ^{[1]} = dA^{[1]}*g'(Z^{[1]})$ $dW^{[1]} = \frac{1}{m}dZ^{[1]} A^{[0]^{T}}$ $db^{[1]} = \frac{1}{m} \sum_{i = 1}^{m} dz^{[1](i)}= \frac{1}{m} np.sum(dZ^{[1]},axis=1, keepdim=True)$

4. L_Layers_NN

L_Layers_NN ： [LINEAR -> RELU] $\times$ (L-1) -> LINEAR -> SIGMOID

For layer $l\;(l=1,2...,L)$,

Forward Propatation And Backward Propagation