Learning Introduction
Learning neural network has been for some time, from the ordinary BP neural network to LSTM long-term memory networks have a certain understanding, but never the structure of the entire system of neural networks record, I believe that these records can help my little deeper understanding the neural network.
Brief introduction
Convolutional neural network (Convolutional Neural Networks, CNN) is a class that contains convolution calculation possessed deep structure of feed-forward neural network (Feedforward Neural Networks), is one of the representatives depth learning algorithm (deep learning) is.
Its main structure is divided into an input layer, a hidden layer and output layer.
In tensorboard, the structure as shown:
For convolutional neural network having an input layer, an output layer of the convolutional neural network normally no different. But hidden layer may be divided into three parts, namely a convolution layer (feature extraction on the input data), pooled layer (feature selection and filtering), fully connected layer (equivalent to the traditional feedforward neural network the hidden layer).
Introduction hidden layer
1, a convolution layer
The convolution of the input image into a set of convolution filter, certain features of the image in each filter activation.
Assuming that a black and white image the size of 5 * 5, like so:
Convolving convolver using the following:
Convolution results:
Convolution process can extract features, convolutional neural network classification is done based on the feature.
In tensorflow, the layer is an important function convolution:
tf.nn.conv2d(input, filter, strides, padding, use_cudnn_on_gpu=None, name=None)，其中：
1, input is an input quantity, shape is [batch, height, width, channels]. ;
2, filter convolution kernel is used;
3, strides is the step size, format [1, step, step, 1], step refers to the long dimension of each step in image convolution;
4, padding: the amount of type string, only one of which is "SAME", "VALID", SAME represents convolution front area of the same image.
2, pooled layer
Cell layer for convolution layer after feature extraction, feature map is passed to the output of the cell layer and feature selection information filtering.
Is the greatest common pool of pooling, the maximum is the maximum value refers to pooling the data taken after being convoluted, characterized in that the maximum extraction.
Pooling assumed window 2X2, step 2.
The original image is:
After pooling is:
In tensorflow, the cell layer is an important function:
tf.nn.max_pool(value, ksize, strides, padding, data_format, name)
1, value: input cell layer, usually in the back contact layer of the cell layer convolution, a Shape [batch, height, width, channels].
2, ksize: pool size of the window, taking a four-dimensional vectors, typically [1, in_height, in_width, 1].
3, strides: convolution and the like, a window dimension in each sliding step, is [1, stride, stride, 1].
4, padding: convolution and the like, may take 'VALID' or 'SAME'.
This connection structure is tensorboard convolutional layer and pooled layer:
3, the whole connecting layer
The same general structure layer fully connected neural network, as shown:
Specific implementation code
Convolution layer, pooling the full implementation code layer connection layer
def conv2d (x, W, step, pad): # for performing the convolution, x is the input value, w is the convolution kernel
return tf.nn.conv2d(x,W,strides = [1,step,step,1],padding = pad)
def max_pool_2X2 (x, step, pad): # for pooling, x is the input value, step is a step number
return tf.nn.max_pool(x,ksize = [1,2,2,1],strides= [1,step,step,1],padding = pad)
def weight_variable (shape): # for obtaining W
initial = tf.truncated_normal (shape, stddev = 0.1) # random values output from the normal distribution in the truncated
return tf.Variable(initial)
def bias_variable(shape): #获得bias
initial = tf.constant (0.1, shape = shape) # value generating plain
return tf.Variable(initial)
def add_layer(inputs,in_size,out_size,n_layer,activation_function = None,keep_prob = 1):
Layer is fully connected for adding #
layer_name = 'layer_%s'%n_layer
with tf.name_scope(layer_name):
with tf.name_scope("Weights"):
Weights = tf.Variable(tf.truncated_normal([in_size,out_size],stddev = 0.1),name = "Weights")
tf.summary.histogram(layer_name+"/weights",Weights)
with tf.name_scope("biases"):
biases = tf.Variable(tf.zeros([1,out_size]) + 0.1,name = "biases")
tf.summary.histogram(layer_name+"/biases",biases)
with tf.name_scope("Wx_plus_b"):
Wx_plus_b = tf.matmul(inputs,Weights) + biases
tf.summary.histogram(layer_name+"/Wx_plus_b",Wx_plus_b)
if activation_function == None :
outputs = Wx_plus_b
else:
outputs = activation_function(Wx_plus_b)
print(activation_function)
outputs = tf.nn.dropout(outputs,keep_prob)
tf.summary.histogram(layer_name+"/outputs",outputs)
return outputs
def add_cnn_layer(inputs, in_z_dim, out_z_dim, n_layer, conv_step = 1, pool_step = 2, padding = "SAME"):
# Convolution for generating cell layer and layer
layer_name = 'layer_%s'%n_layer
with tf.name_scope(layer_name):
with tf.name_scope("Weights"):
W_conv = weight_variable([5,5,in_z_dim,out_z_dim])
with tf.name_scope("biases"):
b_conv = bias_variable([out_z_dim])
with tf.name_scope("conv"):
# Convolution layer
h_conv = tf.nn.relu(conv2d(inputs, W_conv, conv_step, padding)+b_conv)
with tf.name_scope("pooling"):
# Pooling layer
h_pool = max_pool_2X2(h_conv, pool_step, padding)
return h_pool
All codes
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data",one_hot = "true")
def conv2d(x,W,step,pad):
return tf.nn.conv2d(x,W,strides = [1,step,step,1],padding = pad)
def max_pool_2X2(x,step,pad):
return tf.nn.max_pool(x,ksize = [1,2,2,1],strides= [1,step,step,1],padding = pad)
def weight_variable(shape):
initial = tf.truncated_normal (shape, stddev = 0.1) # random values output from the normal distribution in the truncated
return tf.Variable(initial)
def bias_variable(shape):
initial = tf.constant (0.1, shape = shape) # value generating plain
return tf.Variable (initial) Wuxi flow of the hospital http://www.0510bhyy.com/
def add_layer(inputs,in_size,out_size,n_layer,activation_function = None,keep_prob = 1):
layer_name = 'layer_%s'%n_layer
with tf.name_scope(layer_name):
with tf.name_scope("Weights"):
Weights = tf.Variable(tf.truncated_normal([in_size,out_size],stddev = 0.1),name = "Weights")
tf.summary.histogram(layer_name+"/weights",Weights)
with tf.name_scope("biases"):
biases = tf.Variable(tf.zeros([1,out_size]) + 0.1,name = "biases")
tf.summary.histogram(layer_name+"/biases",biases)
with tf.name_scope("Wx_plus_b"):
Wx_plus_b = tf.matmul(inputs,Weights) + biases
tf.summary.histogram(layer_name+"/Wx_plus_b",Wx_plus_b)
if activation_function == None :
outputs = Wx_plus_b
else:
outputs = activation_function(Wx_plus_b)
print(activation_function)
outputs = tf.nn.dropout(outputs,keep_prob)
tf.summary.histogram(layer_name+"/outputs",outputs)
return outputs
def add_cnn_layer(inputs, in_z_dim, out_z_dim, n_layer, conv_step = 1, pool_step = 2, padding = "SAME"):
layer_name = 'layer_%s'%n_layer
with tf.name_scope(layer_name):
with tf.name_scope("Weights"):
W_conv = weight_variable([5,5,in_z_dim,out_z_dim])
with tf.name_scope("biases"):
b_conv = bias_variable([out_z_dim])
with tf.name_scope("conv"):
h_conv = tf.nn.relu(conv2d(inputs, W_conv, conv_step, padding)+b_conv)
with tf.name_scope("pooling"):
h_pool = max_pool_2X2(h_conv, pool_step, padding)
return h_pool
def compute_accuracy(x_data,y_data):
global prediction
y_pre = sess.run(prediction,feed_dict={xs:x_data,keep_prob:1})
correct_prediction = tf.equal(tf.arg_max(y_data,1),tf.arg_max(y_pre,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction,tf.float32))
result = sess.run(accuracy,feed_dict = {xs:batch_xs,ys:batch_ys,keep_prob:1})
return result
keep_prob = tf.placeholder(tf.float32)
xs = tf.placeholder(tf.float32,[None,784])
ys = tf.placeholder(tf.float32,[None,10])
x_image = tf.reshape(xs,[-1,28,28,1])
h_pool1 = add_cnn_layer(x_image, in_z_dim = 1, out_z_dim = 32, n_layer = "cnn1",)
h_pool2 = add_cnn_layer(h_pool1, in_z_dim = 32, out_z_dim = 64, n_layer = "cnn2",)
h_pool2_flat = tf.reshape(h_pool2,[-1,7*7*64])
h_fc1_drop = add_layer(h_pool2_flat, 7*7*64, 1024, "layer1", activation_function = tf.nn.relu, keep_prob = keep_prob)
prediction = add_layer(h_fc1_drop, 1024, 10, "layer2", activation_function = tf.nn.softmax, keep_prob = 1)
with tf.name_scope("loss"):
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=ys,logits = prediction),name = 'loss')
tf.summary.scalar("loss",loss)
train = tf.train.AdamOptimizer(1e-4).minimize(loss)
init = tf.initialize_all_variables()
merged = tf.summary.merge_all()
with tf.Session() as sess:
sess.run (ins)
write = tf.summary.FileWriter("logs/",sess.graph)
for i in range(5000):
batch_xs,batch_ys = mnist.train.next_batch(100)
sess.run(train,feed_dict = {xs:batch_xs,ys:batch_ys,keep_prob:0.5})
if i % 100 == 0:
print(compute_accuracy(mnist.test.images,mnist.test.labels))