Tensorflow-浅层CNN（MNIST数据集）

CNN

卷积神经网络的提出，解决了传统的基于SIFT类型特征提取算法的繁琐问题。原图像可以直接作为输入。
CNN与SIFT一样，对缩放、平移、旋转等畸变具有不变性，有很强泛化性。
* 卷积的权值共享，大幅减少神经网络参数量，防止过拟合（low bias; high var）

* 全连接层由于参数过多、梯度弥散，不利于多层训练

每个卷积会经过以下几个操作
* 图像经过多个不同的卷积核滤波，并加上bias提出局部特征，每个卷积核会映射出一个新的2D图像
* 将卷积核滤波输出进行非线性激活函数处理，如ReLU
* 对激活函数的输出进行池化（降采样），一般使用最大池化，即保留最显著特征，提高畸变容忍能力、
* 最后可以加上LRN(?)，或者Batch Normalization

权值共享

假设图像是1000x1000，则输入为1,000,000x1

全连接层

每一个节点连接1000,000个像素，共1000,000个节点，参数量极大

局部连接

每一个节点与10x10的像素连接，共10x10x1000,000个参数，比全连接缩小10000倍

卷积操作

每一个节点参数一样，且仅仅与核有关系。10x10的卷积层，参数为100

Feather Map

每一个卷积核的输出成为一个Feather Map。通常，前面的卷积层提取简单特征，比如点、线，下一层提取更高阶特征，比如，几何图形，一层一层往下，组合成眼睛、鼻子等五官，最后组合成一张人脸，完成人脸匹配识别。

Tensorflow 实现简单的卷积神经网络

from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

mnist = input_data.read_data_sets("../../mnist/", one_hot = True)
sess = tf.InteractiveSession()

Extracting ../../mnist/train-images-idx3-ubyte.gz
Extracting ../../mnist/train-labels-idx1-ubyte.gz
Extracting ../../mnist/t10k-images-idx3-ubyte.gz
Extracting ../../mnist/t10k-labels-idx1-ubyte.gz

定义初始化函数来简化多次参数的初始化

• 制造噪声来打破对称结构，如截断的正态分布，标准差0.1
• 因为使用ReLU，因此置偏置bias=0.1以避免死亡节点

def weight_variable(shape):
    init_w = tf.truncated_normal(shape=shape, stddev=0.1)
    return tf.Variable(init_w)

def bias_variable(shape):
    init_b = tf.constant(0.1, shape=shape)
    return init_b

定义卷积层、池化层函数简化多次卷积、池化操作

• tf.nn.conv2d是二维卷积函数，卷积参数W[row, col, channel, num_of_convs]
• tf.nn.max_pool是最大池化函数，strides设为横竖步长2，将图片所为原来的一半
• 步长strides[batch, height, width, channels]，batch为图片样本，1代表不放过任何样本；channel=1代表不放过任何通道

# 图片大小不变
def conv2d(x, W):
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')

# 图片大小减半
def max_pool_2x2(x):
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')

定义卷积神经网络的变量

• [-1, 28, 28, 1] 表示，[图片数量不固定， 28x28尺寸，通道数1]

x = tf.placeholder(tf.float32, [None, 784])
y_true = tf.placeholder(tf.float32, [None, 10])
x_image = tf.reshape(x, [-1, 28, 28, 1])

第一个卷积层

• 5x5 size, 1 chanel, 32 convs
• conv2d, bias, ReLU
• max_pooling

W_conv1 = weight_variable([5, 5, 1, 32])   # 32个 5x5x1的卷积
b_conv1 = bias_variable([32])
h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)   # 经过卷积层得到的输出进行relu
h_pool1 = max_pool_2x2(h_conv1)

第二个卷积层

与上一层相似，唯一不同是num_of_convs=64，及提取的特征为64种

W_conv2 = weight_variable([5, 5, 32, 64])             # 32个特征图，因此为32维图像
b_conv2 = bias_variable([64])
h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
h_pool2 = max_pool_2x2(h_conv2)

全连接层

• 前面得到的输出为7x7x64的tensor
• 使用reshape转化为1D的向量，连接节点数1024的全连接层

W_fc1 = weight_variable([7 * 7 * 64, 1024])
b_fc1 = bias_variable([1024])
h_pool2_flat = tf.reshape(h_pool2, [-1, 7 * 7 * 64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)

为了减轻过拟合，Dropout层

keep_prob = tf.placeholder(tf.float32)
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob=keep_prob)

最后，输出到一个softmax层

W_fc2 = weight_variable([1024, 10])
b_fc2 = bias_variable([10])
y_conv = tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2)

二. 定义loss， optimizer

reduction_indices
- 0—–>保持横向量
- 1—–>保持列向量

cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_true * tf.log(y_conv), reduction_indices=[1]))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)

定义评估操作

corrent_pred = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_true, 1))
train_accuracy = tf.reduce_mean(tf.cast(corrent_pred, tf.float32))

训练

tf.global_variables_initializer().run()
with tf.device("/gpu:0"):
    for i in range(20000):
        batch = mnist.train.next_batch(50)
        if i % 100 == 0:
            train_accuracy = accuracy.eval(feed_dict={x:batch[0], y_true:batch[1], keep_prob:0.5})
            print("step %d, training accuracy %g"%(i, train_accuracy))
        train_step.run(feed_dict={x:batch[0], y_true:batch[1], keep_prob:0.5})

step 0, training accuracy 0.04
step 100, training accuracy 0.68
step 200, training accuracy 0.8
step 300, training accuracy 0.88
step 400, training accuracy 0.84
step 500, training accuracy 0.88
step 600, training accuracy 0.88
step 700, training accuracy 0.9
step 800, training accuracy 0.94
step 900, training accuracy 0.92
step 1000, training accuracy 0.9
step 1100, training accuracy 0.92
step 1200, training accuracy 0.96
step 1300, training accuracy 0.9
step 1400, training accuracy 0.98
step 1500, training accuracy 0.96
step 1600, training accuracy 0.98
step 1700, training accuracy 0.96
step 1800, training accuracy 0.94
step 1900, training accuracy 0.98
step 2000, training accuracy 0.92
step 2100, training accuracy 0.96
step 2200, training accuracy 1
step 2300, training accuracy 0.96
step 2400, training accuracy 1
step 2500, training accuracy 1
step 2600, training accuracy 0.94
step 2700, training accuracy 0.98
step 2800, training accuracy 0.92
step 2900, training accuracy 0.98
step 3000, training accuracy 0.98
step 3100, training accuracy 1
step 3200, training accuracy 0.96
step 3300, training accuracy 0.98
step 3400, training accuracy 0.94
step 3500, training accuracy 0.94
step 3600, training accuracy 0.98
step 3700, training accuracy 0.98
step 3800, training accuracy 0.96
step 3900, training accuracy 1
step 4000, training accuracy 1
step 4100, training accuracy 1
step 4200, training accuracy 1
step 4300, training accuracy 0.98
step 4400, training accuracy 1
step 4500, training accuracy 0.96
step 4600, training accuracy 1
step 4700, training accuracy 0.96
step 4800, training accuracy 0.94
step 4900, training accuracy 1
step 5000, training accuracy 0.92
step 5100, training accuracy 0.96
step 5200, training accuracy 0.94
step 5300, training accuracy 0.98
step 5400, training accuracy 1
step 5500, training accuracy 0.96
step 5600, training accuracy 0.98
step 5700, training accuracy 0.98
step 5800, training accuracy 1
step 5900, training accuracy 1
step 6000, training accuracy 0.98
step 6100, training accuracy 1
step 6200, training accuracy 1
step 6300, training accuracy 1
step 6400, training accuracy 1
step 6500, training accuracy 0.98
step 6600, training accuracy 1
step 6700, training accuracy 0.96
step 6800, training accuracy 1
step 6900, training accuracy 0.98
step 7000, training accuracy 1
step 7100, training accuracy 0.98
step 7200, training accuracy 1
step 7300, training accuracy 0.98
step 7400, training accuracy 1
step 7500, training accuracy 1
step 7600, training accuracy 1
step 7700, training accuracy 1
step 7800, training accuracy 1
step 7900, training accuracy 1
step 8000, training accuracy 0.98
step 8100, training accuracy 1
step 8200, training accuracy 0.96
step 8300, training accuracy 0.96
step 8400, training accuracy 0.96
step 8500, training accuracy 0.98
step 8600, training accuracy 1
step 8700, training accuracy 1
step 8800, training accuracy 1
step 8900, training accuracy 1
step 9000, training accuracy 1
step 9100, training accuracy 1
step 9200, training accuracy 0.98
step 9300, training accuracy 1
step 9400, training accuracy 1
step 9500, training accuracy 1
step 9600, training accuracy 1
step 9700, training accuracy 1
step 9800, training accuracy 0.98
step 9900, training accuracy 1
step 10000, training accuracy 1
step 10100, training accuracy 1
step 10200, training accuracy 0.94
step 10300, training accuracy 0.98
step 10400, training accuracy 0.98
step 10500, training accuracy 1
step 10600, training accuracy 1
step 10700, training accuracy 1
step 10800, training accuracy 1
step 10900, training accuracy 1
step 11000, training accuracy 1
step 11100, training accuracy 0.98
step 11200, training accuracy 1
step 11300, training accuracy 0.98
step 11400, training accuracy 0.98
step 11500, training accuracy 0.98
step 11600, training accuracy 0.96
step 11700, training accuracy 1
step 11800, training accuracy 1
step 11900, training accuracy 1
step 12000, training accuracy 1
step 12100, training accuracy 1
step 12200, training accuracy 1
step 12300, training accuracy 1
step 12400, training accuracy 1
step 12500, training accuracy 1
step 12600, training accuracy 1
step 12700, training accuracy 0.98
step 12800, training accuracy 1
step 12900, training accuracy 1
step 13000, training accuracy 1
step 13100, training accuracy 1
step 13200, training accuracy 1
step 13300, training accuracy 1
step 13400, training accuracy 0.98
step 13500, training accuracy 1
step 13600, training accuracy 1
step 13700, training accuracy 1
step 13800, training accuracy 1
step 13900, training accuracy 0.98
step 14000, training accuracy 1
step 14100, training accuracy 1
step 14200, training accuracy 1
step 14300, training accuracy 0.98
step 14400, training accuracy 1
step 14500, training accuracy 1
step 14600, training accuracy 1
step 14700, training accuracy 0.98
step 14800, training accuracy 0.96
step 14900, training accuracy 0.98
step 15000, training accuracy 1
step 15100, training accuracy 1
step 15200, training accuracy 1
step 15300, training accuracy 1
step 15400, training accuracy 1
step 15500, training accuracy 0.98
step 15600, training accuracy 1
step 15700, training accuracy 1
step 15800, training accuracy 0.94
step 15900, training accuracy 1
step 16000, training accuracy 1
step 16100, training accuracy 1
step 16200, training accuracy 0.98
step 16300, training accuracy 1
step 16400, training accuracy 1
step 16500, training accuracy 1
step 16600, training accuracy 1
step 16700, training accuracy 0.98
step 16800, training accuracy 0.96
step 16900, training accuracy 1
step 17000, training accuracy 1
step 17100, training accuracy 0.98
step 17200, training accuracy 1
step 17300, training accuracy 1
step 17400, training accuracy 1
step 17500, training accuracy 1
step 17600, training accuracy 1
step 17700, training accuracy 1
step 17800, training accuracy 1
step 17900, training accuracy 1
step 18000, training accuracy 1
step 18100, training accuracy 0.98
step 18200, training accuracy 1
step 18300, training accuracy 1
step 18400, training accuracy 0.98
step 18500, training accuracy 1
step 18600, training accuracy 1
step 18700, training accuracy 0.98
step 18800, training accuracy 1
step 18900, training accuracy 0.98
step 19000, training accuracy 1
step 19100, training accuracy 1
step 19200, training accuracy 1
step 19300, training accuracy 1
step 19400, training accuracy 1
step 19500, training accuracy 1
step 19600, training accuracy 1
step 19700, training accuracy 0.98
step 19800, training accuracy 1
step 19900, training accuracy 1

print("test accuracy %g"%accuracy.eval(feed_dict = {
    x:mnist.test.images, y_true:mnist.test.labels, keep_prob : 1.0}))

test accuracy 0.9916