DeepLearning With Tensorflow : Deep Mnist and Tensorboard

1. introduction

In this blog, I want firstly to learn some basic functions in tensor flow frame and then to design a convolution network to address one classical problem - mnist recognition. Meanwhile I also would to note some basic usage for my unfamiliar function. Lastly, I would like to learn Tensorboard to visualize the learning phase and explore how to analyze these information.

2. designing network

we firstly import two important modules involved in our project:

import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

Then, we would better define initial functions for weight and bias, which are frequently called in a network:

def weight_variable(shape):
    initial = tf.truncated_normal(shape, stddev=0.1)
    return tf.Variable(initial)


def bias_variable(shape):
    initial = tf.constant(0.1, shape=shape)
    return tf.Variable(initial)

here, 'shape' is a tensor that includes four dimensions.

if we finish these works, we can to design our network, that is our graph. In tensor flow frame, network designing and training are divided into two different phases - graph and session.

myGraph=tf.Graph()
with myGraph.as_default():
    with tf.name_scope('inputsAndLabels'):
        x_raw = tf.placeholder(tf.float32, shape=[None, 784])
        y = tf.placeholder(tf.float32, shape=[None, 10])

    with tf.name_scope('hidden1'):
        x = tf.reshape(x_raw, shape=[-1, 28, 28, 1])  # -1 donates recognition automatically
        W_conv1 = weight_variable([5, 5, 1, 32])
        b_conv1 = bias_variable([32])
        l_conv1 = tf.nn.relu(tf.nn.conv2d(x, W_conv1, strides=[1, 1, 1, 1], padding='SAME') + b_conv1)
        l_pool1 = tf.nn.max_pool(l_conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')

        tf.summary.image('x_input', x, max_outputs=10)
        tf.summary.histogram('W_con1', W_conv1)
        tf.summary.histogram('b_con1', b_conv1)

    with tf.name_scope('hidden2'):
        W_conv2 = weight_variable([5, 5, 32, 64])
        b_conv2 = bias_variable([64])
        l_conv2 = tf.nn.relu(tf.nn.conv2d(l_pool1, W_conv2, strides=[1, 1, 1, 1], padding='SAME')+b_conv2)
        l_pool2 = tf.nn.max_pool(l_conv2, ksize=[1, 2, 2, 1],strides=[1, 2, 2, 1], padding='SAME')

        tf.summary.histogram('W_con2', W_conv2)
        tf.summary.histogram('b_con2', b_conv2)

    with tf.name_scope('fc1'):
        W_fc1 = weight_variable([64*7*7, 1024])
        b_fc1 = bias_variable([1024])
        l_pool2_flat = tf.reshape(l_pool2, [-1, 64*7*7])
        l_fc1 = tf.nn.relu(tf.matmul(l_pool2_flat, W_fc1) + b_fc1)
        keep_prob = tf.placeholder(tf.float32)
        l_fc1_drop = tf.nn.dropout(l_fc1, keep_prob)

        tf.summary.histogram('W_fc1', W_fc1)
        tf.summary.histogram('b_fc1', b_fc1)

    with tf.name_scope('fc2'):
        W_fc2 = weight_variable([1024, 10])
        b_fc2 = bias_variable([10])
        y_conv = tf.matmul(l_fc1_drop, W_fc2) + b_fc2

        tf.summary.histogram('W_fc1', W_fc1)
        tf.summary.histogram('b_fc1', b_fc1)

    with tf.name_scope('train'):
        cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_conv, labels=y))
        train_step = tf.train.AdamOptimizer(learning_rate=1e-4).minimize(cross_entropy)
        correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y, 1))
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

        tf.summary.scalar('loss', cross_entropy)
        tf.summary.scalar('accuracy', 

• myGraph=tf.Graph(): design a new graph and return a object.
  
• myGraph.as_default(): overlay the default graph.
• with tf.name_scope('name'): used to administrate operations in graph, and return a context manager which named by scope_name. The graph will maintain the stack of name_space, so we can define different operations  or children namespace in each name_space. So all in all, using it is to administrate the graph orderly, and in case of conflicts among operations.
• x_raw = tf.placeholder(tf.float32, shape=[None, 784]) &
  y = tf.placeholder(tf.float32, shape=[None, 10] : when we design network graph, we do not know the batch of input images and labels. so we adopt place holder. The first parameter of shape - none- is the size of batch, which can be assigned automatically once the session runs. 784 = 28*28 is the size of input image. 10 is the image is classed into ten classes.
•  x = tf.reshape(x_raw, shape=[-1, 28, 28, 1]): in order to make sure dimension match, when input batch is feed to first layer we should reshape the shape of input tensor. Then common shape is [-1, in_height,  in_width, in_channel]. we should notice "-1", which means the program can recognize the size of batch once the session runs.
• tf.nn.conv2d(input, filter, strides, padding, use_cudnn_on_gpu=None, data_format=None, name=None)
  input = [batch, in_height, in_width, in_channel]
  filter = [filter_height, filter_width, in_channel, out_channel]
  strides = [batch_s, imheight_s, imwidth_s, imchannel_s]  #stride
  padding = 'VALID' or 'SAME'
• tf.nn.relu(features, name = None)
• tf.nn.max_pool(input, ksize, strides, padding, name=None)
  input = [batch, in_height, in_width, in_channel]
  ksize =[batch_d, imheight_d, imwidth_d, imchannel_d ] #down sample
  strides = [batch_s, imheight_s, imwidth_s, imchannel_s]  #stride
  padding = 'VALID' or 'SAME'
• tf.nn.dropout(x, keep_prob, noise_shape=None, seed=None,name=None) : always occurs in full connection layer.
  
•  cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits( ogits=y_conv, labels=y)): measure function
  and then;
  train_step = tf.train.AdamOptimizer(learning_rate=1e-4).minimize()

3. run session and train

Run a session:

with tf.Session(graph=myGraph) as sess:
    sess.run(tf.global_variables_initializer())
    saver = tf.train.Saver()

    merged = tf.summary.merge_all()
    summary_writer = tf.summary.FileWriter('./mnistEven/', graph=sess.graph)

    for i in range(3001):
        batch = mnist.train.next_batch(100)
        sess.run(train_step, feed_dict={x_raw: batch[0], y: batch[1], keep_prob: 0.8})
        if i % 100 == 0:
            train_accuracy = accuracy.eval(feed_dict={x_raw: batch[0], y: batch[1], keep_prob: 0.8})
            print('step %d training accuracy:%g' % (i, train_accuracy))

            summary = sess.run(merged, feed_dict={x_raw: batch[0], y: batch[1], keep_prob: 0.8})
            summary_writer.add_summary(summary, i)

4. test

Lastly, test the network

test_accuracy = accuracy.eval(feed_dict={x_raw: mnist.test.images, y: mnist.test.labels, keep_prob: 1.0})
    print('test accuracy:%g' % test_accuracy)
    saver.save(sess, save_path='mnist_model/mnistmodel', global_step=1)

5. visualizing with tensorflow

python /home/shenziheng/anaconda3/lib/python3.6/site-packages/tensorflow/tensorboard/tensorboard.py --logdir=/home/shenziheng/PycharmProjects/advance_mnist/mnistEven