HongYiLee Backpropagation Notes

标签： Notes DeepLearning Backpropagation

Introduction of Deep Learning

1. Given a network structure, mean we define the function set.

给定一个神经网络的结构，那么就是定义了一个函数集合。然后我们要做的就是去找到一个最好的best function。

What does a Nerual network do?

y = f (x) = σ (W^{L} . . . σ (W^{2} (σ (W^{1} + b^{1}) + b^{2}) + . . . + b^{L})

$y = f(x)=\sigma(W^L...\sigma(W^2(\sigma(W^1+b^1)+b^2)+...+b^L)$
So we can using parallel computing techniques to speed up matrix operation.
对于矩阵运算，我们可以使用 GPU的加速运算能力。

2. Define the goodness or badness of a function. We call it Loss Function.

3. Pick the function minimize the Loss Function.

Backpropagation反向传播

L (θ) = \sum_{n = 1}^{N} l^{n} (θ)

$L(\theta)=\sum_{n=1}^{N}l^n(\theta)$

\frac{\partial L (θ)}{\partial w} = \sum_{n = 1}^{N} \frac{\partial l (θ)}{\partial w}

$\frac{\partial L(\theta)}{\partial w} = \sum_{n=1}^{N}\frac{\partial l(\theta)}{\partial w}$
So we need just to compute

\frac{\partial l (θ)}{\partial w}

$\frac{\partial l(\theta)}{\partial w}$ .

\frac{\partial l (θ)}{\partial w} = \frac{\partial l (θ)}{\partial z} \frac{\partial z}{\partial w}

$\frac{\partial l(\theta)}{\partial w} = \frac{\partial l(\theta)}{\partial z}\frac{\partial z}{\partial w}$
So we have Forward pass:compute

\frac{\partial z}{\partial w}

$\frac{\partial z}{\partial w}$ .
and Backward pass: compute

\frac{\partial l (θ)}{\partial z}

$\frac{\partial l(\theta)}{\partial z}$ .

Forward pass:
$\frac{\partial z}{\partial w_{1}} = x_{1}$ $\frac{\partial z}{\partial w_1}=x_1$
$\frac{\partial z}{\partial w_{2}} = x_{2}$ $\frac{\partial z}{\partial w_2}=x_2$
$. . .$ $...$
$\frac{\partial z}{\partial w_{i}} = x_{i}$ $\frac{\partial z}{\partial w_i}=x_i$
Backward pass:
Let $a=\sigma(z)$ , $\frac{\partial l}{\partial z} = \frac{\partial a}{\partial z} \frac{\partial l}{\partial a}$ $\frac{\partial l}{\partial z}=\frac{\partial a}{\partial z}\frac{\partial l}{\partial a}$
$\frac{\partial a}{\partial z} = σ^{^{'}} (z)$ $\frac{\partial a}{\partial z}=\sigma^{'}(z)$
$\frac{\partial l}{\partial a} = \frac{\partial z^{^{'}}}{\partial a} \frac{\partial l}{\partial z^{^{'}}} + \frac{\partial z^{^{″}}}{\partial a} \frac{\partial l}{\partial z^{^{″}}}$ $\frac{\partial l}{\partial a} = \frac{\partial z^{'}}{\partial a}\frac{\partial l}{\partial z^{'}} +\frac{\partial z^{''}}{\partial a}\frac{\partial l}{\partial z^{''}}$ So this is Chain Rule.