Deeplearning.ai recent study on the neural network, did some tests, there are a number of insights, record, learn to share:
First, the neural networks
Neural Networks: Neural Networks, a similar brain synaptic coupling mathematical model of the structure of the application information processing
1, neurons
Neurons can feel the changes in the environment, and passes the information to the next neuron; the human brain has 86 billion neurons
2, the theory of evolution
600 million years ago, life on Earth began to appear, but can not do anything, the problem is that we are not nervous
no nerves, can not move, can not process any information, there is only silence, until death
later, there have been jellyfish, jellyfish found We must have nerves, so it has the world's first set of the nervous system - the neural network
important information jellyfish neural network so that it can be collected from around the world, such as where there are predators and food, and then pass this information to various parts of the body
which means that jellyfish can respond to changes in the environment to improve the chances of survival rather than just floating aimlessly, pray for good luck
3, the depth of learning
Introduce a pioneer: In 2006, Geoffrey Hinton, who first proposed the depth of learning, using the "depth" of the word.
The introduction of a called greedy layer wise pre-training policy, other researchers have found that this is very effective for deeper training of the neural network, which led to a wave of enthusiasm.
But there is a view that the industry - and they do so in order to apply for research funding
4, Andrew Ng views
Great God views are contradictory time, of course, this must be a reason:
This is the depth of learning, Ng depth study of the effect of the practice, recognizing the importance of basic "neural network" deep learning. This is the reason he focuses on the neural networks in the back.
Second, the rise of the depth of learning
The reason: data, computing power, algorithm
(1) Algorithm: nearly 20 years of continuous innovation, a lot of innovation in order to calculate more quickly (such as sigmoid function evolved into Relu function)
(2) sigmoid function at the end, the gradient is very small, approximately 0, so that when the gradient descent method, very little change in a parameter, the training is very slow
(3) fast computing is important: idea - code - experiment, quickly verify iteration
Traditional machine learning methods, such as vector machine SVM, when dealing with small amounts of data, with the increasing amount of data, the performance will increase; but when a certain amount of data will encounter bottlenecks;
then how to achieve good "performance of the performance" mean : the first large-scale neural network needs, the second requires a lot of data
affect the training data set size on performance:
1, when the data set is small: svm performance as well, is likely to be the skills manual design components of relatively high
2, when large data sets: neural network algorithms will lead other stable
Third, industrial applications
(1) supervised learning
exaggerate the depth of sound never stopped learning, but currently produces only supervised learning machine learning in economic benefits
(2) unstructured data
scene early field is generated economic benefits of structured data; need to thank depth learning and neural networks, so that people can make progress in image, voice field (unstructured data)
Common structure Fourth, the neural networks
We should be more familiar with this one, only for review shows:
Depth neural network architecture: DNN, CNN, RNN, LSTM, BLSTM
Fifth, the application of neural network learning system depth is how it works
For example, a price forecast:
Known housing area, it is desirable forecast prices of the house area - price: to fit by a linear function;
magic neural network comprising: a long enough (x, y), can be calculated precisely mapping function.
Our mission is: for a given x and y, find a fitting function
Factors Below we expand housing prices: the area, room type, location, surrounding facilities
X: x1, x2, x3, x4 ...... when conditions become more input: n-dimensional vector
of the hidden layer: If you just do a linear mapping, output forever X is a linear transformation, expression is not enough
to activate the function: nonlinear transformation function, the purpose is to extract a higher dimensional feature, See more
meaningful output in order to ensure: a binary classification application, do the output layer normalization ( softmax function)
Six, demo verification
Our experimental environment based on the following: python + tensorflow, verify the prediction model can work correctly, the tool environment:
1，tensorflow
Google's open source depth learning tools
second-generation learning platform to enhance the efficiency of work (on the first-generation system DistBelief performance scalability is very good, but in achieving the desired flexibility for research)
into the CPU and GPU two versions of
the underlying C ++ to write front-end support for C ++, Python, internal support has begun to Go
supports to run on various hardware platforms, including mobile phone, PC, server clustering
tool: Tensorboard, web interface, debugging tools
2，Anaconda
Anaconda is the open source release of the Python language for large-scale data processing, predictive analysis, and scientific computing, simplifying the deployment and management packs
official website: https: //www.anaconda.com/download/
pre-loaded with a lot of third-party libraries, such as pip
recommend Python 3.6, containing scientific computing environment, Python 2.x package does not
pip command to install tensorflow Note: after installing anaconda, use the command pip tool installation tensorflow of: pip --proxy 10.14.87.100:8080 install tensorflow (cpu installed version); in http://auth-proxy.oa.com/DevNetTempVisit.aspx need to apply for temporary permission to extranet
3，Jupyter Notebook
Jupyter Notebook (formerly known IPython notebook) is an interactive notebook, run support more than 40 programming languages.
Processing for data analysis - calculation - analysis of
Python package brings a wealth of third-party support, do not need to repeat yourself and create the wheel
is gradually rich features, such as multiple cell's Cut / Paste, PPT show, cell tag
4, neural network design
The simplest training a neural network
has only one neuron
input: X (1, n)
Output: Y (1, n)
Goal: iterative training through training, so that parameters (w, b) fitting function converges
In a relatively deep neural network parameters w, b L is calculated in accordance with the result of each forward propagation,
Derivative (such fastest gradient descent) and then reverse spread, layer by layer update (reverse),
Complete one iteration (or called training) Sigma activation function, m samples;
Gradient descent is required minimum value, instead of the minimum value;
Note: step selection is critical, step up not too extreme points even divergent, convergent step short lead time is too long.
Parameter update:
General training process Summary:
To summarize: work steps roughly depth study of the neural network system is divided into:
1, forward propagation: Calculated loss function
2, backpropagation network parameter update
3, 1/2 links repeated continuously iterative implementation of
Attachment: another demo, Mnist class prediction handwritten figures
Feel unsatisfied above experiments, a look MNIST experiment:
MNIST data set from the National Institute of Standards and Technology, National Institute of Standards and Technology ( NIST) training set (training set) comprised of a digital from 250 different human handwriting. of which 50% are high school students, 50% of the staff from the census Bureau (the census Bureau) the test set (test set) is the same proportion of handwritten digital data.
MNIST data set available in http://yann.lecun.com/exdb/mnist/, which comprises four parts:
Training Images SET: Train-Images-idx3 does-ubyte.gz (60,000 samples)
Training SET Labels: train-labels-idx1-ubyte.gz ( 60,000 tags)
the Test Images SET: Images-T10K-idx3 does-ubyte.gz (10,000 samples)
the Test SET labels: labels-T10K-the idx1-ubyte.gz (10,000 tags)
We can visualize these pictures:
6 digital example, about long like this:
Our task is to train the neural network to recognize handwritten numbers.
The following is the actual code under test mnist jupyter environment, particularly adjustable parameters, are interested can download the reference data set after commissioning:
! # / usr / bin / python3.5
# - * - Coding: UTF-. 8 - * -
to achieve the above-described idea # code as follows:
# 1 reads mnist handwritten data;.
# 2 data values from [0,. 1] range of floating-point format is converted to black and white (black background is from 0, the outlook for the 255- white);
. # 3 based on the content mnist.train.labels generate a numerical index, which is the establishment of each picture and they represent associated digital, thereby creating a corresponding save directory;
# 4 loop through mnist.train.images, to each image pixel is assigned to a class instance PIL Image, and then call the save method of the Image class to save the picture in the third the corresponding directory created in step.
# The result: generating mnist_digits_images directory in the current directory, which contains the true picture of mnist
Import os
Import tensorflow AS TF
from tensorflow.examples.tutorials.mnist Import input_data
from PIL Import Image
# declare picture width and height of
rows = 28
cols = 28
# number of pictures to be extracted
images_to_extract = 8000
to save the current directory in the path #
save_dir = "./mnist_digits_images"
# reads data mnist
= input_data.read_data_sets MNIST ( "/ ai-the Data / MNIST", one_hot = False)
# Create Session
sess = tf.Session ()
# get the total picture
shape = sess.run (tf.shape (mnist.train.images) )
Shape = IMAGES_COUNT [0]
pixels_per_image Shape = [. 1]
# Number of acquisition tab
Shape = sess.run (tf.shape (mnist.train.labels))
labels_count Shape = [0]
# is a two-dimensional sheet mnist.train.labels amount, for purposes of subsequently generated digital image directory name, it is necessary one-dimensional (later found as long as the one_hot attribute data set to False, mnist.train.labels itself is a one-dimensional)
#labels = sess.run (tf.argmax (mnist.train.labels, 1))
Labels = mnist.train.labels
# to check whether the expected format data sets
iF (IMAGES_COUNT == labels_count) and (shape.size == 1):
Print ( "data set contains a total of% s pictures, and tags% s "% (images_count, labels_count))
print ( "% s each image comprising pixels"% (pixels_per_image))
print ( "Data Type:% s"% (mnist.train.images.dtype))
# mnist image data value range is [0,1] , you need to be extended to [0,255], so that the human eye
IF mnist.train.images.dtype == "float32":
Print ( "prepare the data type from [0,1] into binary [0,255] ..." )
for I in Range (0, images_to_extract):
for n-in Range (pixels_per_image):
IF mnist.train.images [I] [n-] = 0:!
mnist.train.images [I] [n-] = 255
# due a large number of data sets pictures, the conversion may take a lot of time, it is necessary to print progress of the conversion
IF ((i + 1)% 50) == 0:
Print ( "image floating-point value expansion schedule: converted% s Zhang, a total of must be converted to% s Zhang "% (i + 1, images_to_extract))
# create a digital image save directory
for i in the Range (10):
dir ="% s /% s / "% (save_dir, i)
IF not os. path.exists (dir):
Print ( "Catalog""% S" "does not exist! Is automatically created ..."% dir)
os.makdirs (the dir)
# python by image processing library, generate picture
indices = [0 for X in Range (0, 10)]
for I in Range (0, images_to_extract):
IMG Image.new = ( "L", ( cols, rows))
for Range in m (rows):
for n-in Range (cols):
img.putpixel ((n-, m), int (mnist.train.images [I] [n-m * cols +]))
# path to generate a corresponding stored digital label image represented by the
digit for Labels = [I]
path = "% S /% S /% s.bmp"% (save_dir, Labels [I], indices [digit for])
indices [digit for ] + = 1
img.save (path)
# due to the large number of picture data sets, the save process may take a lot of time, it is necessary to save the print progress
IF ((i + 1)% 50) == 0:
Print ( "picture save progress: saved% s Zhang, Zhang totaling save% s "% (i + 1, images_to_extract))
the else:
Print (" inconsistent with the number of images and the number of labels ")!