Machine Learning |吴恩达 公式总结（2）【傻瓜入门版】（未完稿）

Neural Networks Learning

Neural network model

Forward propagation:

cost function

notation:
$a_{i}^{(j)}$ = “activation” of unit i in layer j
$\Theta^{(j)}$ = matrix of weights controlling function mapping from layer j to layer j+1
$h_{\theta}(x^{(i)})_{k}=a_{k}^{(3)}$ is the activation (output value) of the k-th output unit
$y=\begin{bmatrix} y_{k=1} \quad 1 \\ y_{k=2} \quad 0\\ y_{k=3} \quad 0 \\ \vdots \\ y_{k=10} \quad0 \end{bmatrix},\begin{bmatrix} 0 \\ 1\\0 \\ \vdots \\0 \end{bmatrix},\dots or \begin{bmatrix} 1 \\ 0\\0 \\ \vdots \\1 \end{bmatrix}$ 是one-hot编码。
【备注：one-hot编码的优点：不同数字如（0，1，….9）之间的欧式距离是相等的。】
$y_{k}^{(i)}=$第i个样本的的第j个输出单元。the k-th output unit of the i-th example
$L=$网络层的总数，包括输入层与输出层。the number of the layers,include input layer and output layer

$$J(\theta)=-\frac{1}{m}\sum_{i=1}^{m}\sum_{k=1}^{k}\bigg [ y_{k}^{(i)}log((h_{\Theta}(x^{(i)}))_{k})+(1-y_{k}^{(i)})log(1-(h_{\Theta}(x^{(i)}))_{k}) \bigg] + \frac{\lambda}{2m}\sum_{l=1}^{L-1}\sum_{i=1}^{sl}\sum_{j=1}^{s_{(l+1)}}(\Theta_{j,i}^{(l)})^2$$

backpropagation algorithm

Backpropagation:

Training set {$(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),\dots ,(x^{(m)},y^{(m)})$} .m examples.
Set $\Delta_{ij}^{(l)}=0$ (for all l,i,j) ,used to comput $\frac{∂}{∂\Theta_{i,j}^{(l)}}J(\Theta),$ Cycle accumulation
For training example t=1 to m:
1. Set $a^{(1)}:=x^{(t)}$
2.执行前向传播，分别计算 $a^{(l)}$ for $l=2,3,\dots ,L$
3.用$y^{(i)}$，通过$\delta ^{(L)} = a ^{(L)} - y^{(t)}$计算输出层的损失。
4.Compute $\delta ^{(L-1)},\delta ^{(L-2)},\dots ,\delta ^{(2)}.\quad \delta ^{(l)}=\big((\Theta^{(l)})^{T} \delta ^{(l+1)} \big).*a^{(l)}.*\big( 1-a^{(i)}\big)\quad \quad$ //$g'(z^{(l)})=a^{(l)}.*\big( 1-a^{(i)}\big)$
5.$\Delta_{i,j}^{(l)}:=\Delta_{i,j}^{(l)}+a_{j}^{(l)}\delta_{i}^{(l+1)}$ or with vectorization,$\Delta^{(l)}:=\Delta^{(l)}+\delta^{(l+1)}(a^{(l)})^T$.
ENDFOR

hense we update our new $\Delta$matrix.

• $D_{i,j}^{(l)}:=\frac{1}{m}\big(\Delta_{i,j}^{(l)}+\lambda\Theta_{i,j}^{(l)} \big),if j \neq 0$
• $D_{i,j}^{(l)}:=\frac{1}{m}\Delta_{i,j}^{(l)},if j =0$
  Thus we get $\frac{∂}{∂\Theta_{i,j}^{(l)}}J(\Theta)=D_{i,j}^{(l)}$

partial derivative work processes

$g'(z^{(l)})=a^{(l)}.*\big( 1-a^{(i)}\big)$推导过程：
已知：$g(z^{(l)})= \frac{1}{1+e^{-z^{(l)}}}=a^{(l)}$
$g'(z^{(l)})=\big(sigmoid(z^{(l)})\big)'$
$= \big(\frac{1}{1+e^{-z^{(l)}}}\big)'$
$= -1\cdot \frac{1}{(1+e^{-z^{(l)}})^2}\cdot(1+e^{-z^{(l)}})'$
$= \frac{-1}{(1+e^{-z^{(l)}})^2}\cdot(e^{-z^{(l)}})'$
$= \frac{-1}{(1+e^{-z^{(l)}})^2}\cdot e^{-z^{(l)}} \cdot (-z^{(l)})'$
$= \frac{-1}{(1+e^{-z^{(l)}})^2}\cdot e^{-z^{(l)}} \cdot -1$
$= \frac{(e^{-z^{(l)}}+1)-1}{(1+e^{-z^{(l)}})^2}$
$= \frac{1}{1+e^{-z^{(l)}}} - \frac{1}{(1+e^{-z^{(l)}})^2}$
$= g(z^{(l)}) \cdot (1-g(z^{(l)}))$
$= a^{(l)} \cdot (1-a^{(l)})$

反向传播部分的计算比较繁琐，特别作了一张简化了的网络示意图。实际效果与课程中样例是一致的。

$a^{(1)}=x^{(i)}$是样本的输入层，图中示意X有两个特征，另外增加一个偏置量$a_{0}^{(1)}$
$a^{(2)}$层是隐藏层，有三个节点，$a^{(1)}*\theta^{(1)}$后输出$z^{(2)}$,$z^{(2)}$再经过激活函数sigmoid输出$a^{(2)}$。隐藏层添加一个偏置量$a_{0}^{(2)}$
$a^{(3)}$是输出层，也有激活的过程，输出的$a_{1}^{(3)}=y_{1}$对应公式里的$(h_{\theta}(x^{(i)}))_{k=1},a_{2}^{(3)}=y_{2}$对应公式里的$(h_{\theta}(x^{(i)}))_{k=2}$,$h_\theta(x^{(i)})=[y_{1}^{(i)} ,y_{2}^{(i)}]$
回顾一下神经网络的损失函数：

$$J(\theta)=-\frac{1}{m}\sum_{i=1}^{m}\sum_{k=1}^{k}\bigg [ y_{k}^{(i)}log((h_{\Theta}(x^{(i)}))_{k})+(1-y_{k}^{(i)})log(1-(h_{\Theta}(x^{(i)}))_{k}) \bigg] + \frac{\lambda}{2m}\sum_{l=1}^{L-1}\sum_{i=1}^{sl}\sum_{j=1}^{s_{(l+1)}}(\Theta_{j,i}^{(l)})^2$$
计算损失：
已知前提：
1) $h_{\theta}(x^{(i)})$就是输出层的输出结果，那么 $h_{\theta}(x^{(i)})=a^{(L)}=sigmoid(z^{(L)})$
2) $z^{(L)}=\theta^{(L-1)} \cdot a^{(L-1)}$
3) $J(\Theta)$考虑单条训练样本省略m，将输出值作为整体计算，即将 $y_{k}$表达为 $y$,同时省略正则项可以表示为： $J(\theta)=-\bigg [ y\cdot log(a^{(L)})+(1-y)log(1-a^{(L)}) \bigg]$。
4) 输出层的误差定义为 $\delta^{(L)}= a^{(L)}-y$
$\theta^{(L)}$的梯度： $\frac{∂J(\Theta)}{∂\theta^{(L)}}$,在上图的3层网络中 $\theta^{(L)}=\theta^{(2)}$
计算该梯度目的是为了计算 $\theta^{(L)}:=\theta^{(L)}-\frac{∂J(\Theta)}{∂\theta^{(L)}}$
$\frac{∂J(\Theta)}{∂\theta^{(L)}}=\frac{∂J(\Theta)}{∂a^{(L)}} \cdot \frac{∂a^{(L)}}{∂z^{(L)}} \cdot \frac{∂z^{(L)}}{∂\theta^{(L)}}$
$\frac{∂J(\Theta)}{∂a^{(L)}} =\frac{a^{(L)}-y}{(1-a^{(L)}) \cdot a^{(L)}}$
$\frac{∂a^{(L)}}{∂z^{(L)}} = a^{(L)} \cdot (1-a^{(L)})$
$\frac{∂z^{(L)}}{∂\theta^{(L)}} = a^{(L-1)}$

合并：$\frac{∂J(\Theta)}{∂\theta^{(L)}}=a^{(L-1)} \cdot (a^{(L)}-y)$
(非输出层)
$\delta^{(l)}=a^{(l)}\cdot \delta^{(l+1)}$