Neural Network

1. Model Summary

At a very simple level, neurons are basically computational units that take inputs (dendrites) as electrical inputs (called “spikes”) that are channeled to outputs (axons). In our model, our dendrites are like the input features $x_1\cdots x_n$ and the output is the result of our hypothesis function. In this model our $x_{0}=1$ input node is sometimes called the “bias unit.” It is always equal to 1. In neural networks, we use the same logistic function as in classification, $\frac{1}{1 + e^{-\theta^Tx}}$ , yet we sometimes call it a sigmoid (logistic) activation function. In this situation, our “theta” parameters are sometimes called “weights”.
Visually, a simplistic representation looks like:

[\begin{matrix} x_{0} \\ x_{1} \\ x_{2} \end{matrix}] \to [] \to h_{θ} (x)

$\begin{bmatrix} x_0 \\ x_1 \\ x_2 \end{bmatrix} →[ \qquad ]→h_θ(x)$

three layers: input layer / hidden layer / output layer

a_{i}^{(j)}

$a_{i}^{(j)}$ : activation unit i in layer j

Θ^{(j)}

$\Theta^{(j)}$ : Matrix that controls function mapping from j-th layer to (j+1)-th layer
If layer j has

s_{j}

$s_{j}$ units, layer j+1 has

s_{j + 1}

$s_{j+1}$ units, then size of

Θ^{(j)}

$\Theta^{(j)}$ is

s_{j + 1} * (s_{j} + 1)

$s_{j+1}*(s_{j}+1)$

L

$L$ : Number of Layers

s_{l}

$s_l$ : Number of units in l-th layer
Number of Inputs: the dimension of features in

x^{(i)}

$x^{(i)}$
Binary Classification: 1 output unit
K-classes Classification: K output unit

2. Forward Propagation

1) Add $a_x^{(0)}=1$ first
2) $z_{x+1}=\Theta^{(x)}a_x$
3) $a_{x+1}=g(z_{x+1})$ — g(x) : Sigmoid

3. Cost Function

Cost Function
Excluding Bias Term

4. Backpropagation Algorithm

$\delta_j^{(l)}$ error of node j in layer l, then

δ^{(L)} = a^{(L)} - y δ^{(i)} = (Θ^{(i)})^{T} δ^{(i + 1)} . * g^{'} (z^{(i)}) (i! = L, i! = 1)

$\delta^{(L)}=a^{(L)}-y\\ \delta^{(i)}=(\Theta^{(i)})^T\delta^{(i+1)}.*g'(z^{(i)}) \qquad (i!=L,i!=1)$
where

g^{'} (z^{(i)}) = a^{(i)} . * (1 - a^{(i)})

$g'(z^{(i)})=a^{(i)}.*(1-a^{(i)})$
这里写图片描述

One thing to note: use one training set to train the model at one time!

5. Unrolling Parameters

Enroll them to vectors/Get back:
这里写图片描述

6.Gradient Checking

这里写图片描述

When learning, turn off gradient checking!!!

7. Random Initialization