Returns the neural network directory
Previous: depth articles - neural network (f) elaborate data enhancement and fine-tuning
In this section, elaborate neural network handwritten numeric code demonstrates
Download project on github: mnist_pro project
6. The Code Project presentation
(1 Introduction
Although we have learned a lot in front of the depth of neural network theory, probably, also know is that training and testing, however, the project how to do it? Estimates that there are still many people forced to look ignorant. How ignorant forced to turn theory into code, to build the project. Below, I will use a simple handwritten digital project, to explain to you briefly.
(2) First, the project needs to be clear
Project requirements, we want a handwritten Arabic numerals 0 to 9 pictures identified. For example, the above invoice number (the first to do digital handwriting recognition, in 1989, a Bank of America hired chiefs wrote, was with convolutional neural network technology LeNet-5 write its recognition rate than depth nerve DNN is better. behind, will explain convolution neural network for everyone. At that time the project is to identify the check signing of the above numbers). Network training, of course inseparable from the data, so we have to download the data, the data is already uploaded for everyone to Baidu cloud disk: link: https: //pan.baidu.com/s/13OokGc0h3F5rGrxuSLYj9Q extraction code: qfj6.
(3). Building project
Project is structured as follows:
The above model, I get random training 10 epoch accuracy: 0.9614. Before I ran 100 epoch, the accuracy to more than 0.98.
(4) depend on the environment and README.md
Depend on the environment:
pip install numpy==1.16
pip install easydict
conda install tensorflow-gpu==1.13.1 # 建议不要用 2.0 版本的 tf，坑多tensorflow installed, my previous blog a detailed explanation: broken point articles --tensorflow gpu version installed if not installed, you can see how to install.
README.md file
# mnist_pro
DNN 手写数字预测 2020-02-06
- 项目下载地址：https://github.com/wandaoyi/mnist_pro
- 请到百度云盘下载项目所需要的训练数据：
- 链接：https://pan.baidu.com/s/13OokGc0h3F5rGrxuSLYj9Q 提取码：qfj6
## 参数设置
- 在训练或预测之前，我们要先进行参数设置
- 打开 config.py 文件，对其中的参数或路径进行设置。
## 训练模型
- 运行 mnist_train.py ，简单操作，右键直接 run
- 训练效果如下：
- acc_train: 0.90625
- y_perd: [7 2 1 0 4]
- y_true: [7 2 1 0 4]
- epoch: 10, acc_test: 0.9613999724388123
- epoch: 10, acc_test_2: 0.9606000185012817
- 下面是随意训练的效果，如果想效果好，可以多训练多点epoch
- 也可以自己添加 early-stopping 进去，不麻烦的
## 预测
- 运行 mnist_test.py ，简单操作，右键直接 run
- 运行后，部分预测结果会打印在控制台上
- 预测效果如下：
- 预测值: [7 2 1 0 4]
- 真实值: [7 2 1 0 4]
## tensorboard 日志
- 使用 tensorboard 的好处是，这个日志是实时的，可以一边训练一边看效果图。
- 在 cmd 命令窗口，输入下面命令：
- tensorboard --logdir=G:\work_space\python_space\pro2018_space\wandao\mnist_pro\logs\mnist_log_train --host=localhost
- 在 --logdir= 后面是日志的文件夹路径，
- 在 --host= 是用来指定 ip 的，如果不写，则只能电脑的地址，而不能使用 localhost
- 在 谷歌浏览器 上打开 tensorboard 日志: http://localhost:6006/
-
-
-
-
-
- 测试日志也是这样打开来看，就不详细去说了。
-
- 关于其他的 ROC 曲线 或 mAP 等，这里就没做这些操作。以后的项目，再操作一番就是了。
The following files or code, which, well commented
(5) The configuration file config.py
#!/usr/bin/env python
# _*_ coding:utf-8 _*_
# ============================================
# @Time : 2020/02/05 13:51
# @Author : WanDaoYi
# @FileName : config.py
# ============================================
from easydict import EasyDict as edict
import os
__C = edict()
cfg = __C
# common options 公共配置文件
__C.COMMON = edict()
# windows 获取文件绝对路径, 方便 windows 在黑窗口 运行项目
__C.COMMON.BASE_PATH = os.path.abspath(os.path.dirname(__file__))
# # 获取当前窗口的路径, 当用 Linux 的时候切用这个，不然会报错。(windows也可以用这个)
# __C.COMMON.BASE_PATH = os.getcwd()
__C.COMMON.DATA_PATH = os.path.join(__C.COMMON.BASE_PATH, "dataset")
# 隐藏层的输出节点数
__C.COMMON.HIDDEN_1 = 300
__C.COMMON.HIDDEN_2 = 100
__C.COMMON.HIDDEN_3 = 50
# 训练配置
__C.TRAIN = edict()
# 学习率
__C.TRAIN.LEARNING_RATE = 0.01
# batch_size
__C.TRAIN.BATCH_SIZE = 32
# 迭代次数
__C.TRAIN.N_EPOCH = 10
# 模型保存路径, 使用相对路径，方便移植
__C.TRAIN.MODEL_SAVE_PATH = "./checkpoint/model_"
# dropout 的持有量，0.7 表示持有 70% 的节点。
__C.TRAIN.KEEP_PROB_DROPOUT = 0.7
# 测试配置
__C.TEST = edict()
# 测试模型保存路径
__C.TEST.CKPT_MODEL_SAVE_PATH = "./checkpoint/model_acc=0.961400.ckpt-10"
# 日志配置
__C.LOG = edict()
# 日志保存路径，后面会接上 train 或 test: 如 mnist_log_train
__C.LOG.LOG_SAVE_PATH = "./logs/mnist_log_"
(6). Common.py common code
#!/usr/bin/env python
# _*_ coding:utf-8 _*_
# ============================================
# @Time : 2020/02/05 14:09
# @Author : WanDaoYi
# @FileName : common.py
# ============================================
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
from config import cfg
import numpy as np
class Common(object):
def __init__(self):
# 数据路径
self.data_file_path = cfg.COMMON.DATA_PATH
self.hidden_01 = cfg.COMMON.HIDDEN_1
self.hidden_02 = cfg.COMMON.HIDDEN_2
self.hidden_03 = cfg.COMMON.HIDDEN_3
pass
# 读取数据
def read_data(self):
# 数据下载地址: http://yann.lecun.com/exdb/mnist/
mnist_data = input_data.read_data_sets(self.data_file_path, one_hot=True)
train_image = mnist_data.train.images
train_label = mnist_data.train.labels
_, n_feature = train_image.shape
_, n_label = train_label.shape
return mnist_data, n_feature, n_label
# bn 操作
def layer_bn(self, input_data, is_training, name, momentum=0.999, eps=1e-3):
"""
:param inputdata: 输入数据
:param is_training: 是否是训练 ，True 为训练
:param name: 名字
:param momentum: 动量因子
:param eps:
:return:
"""
return tf.layers.batch_normalization(inputs=input_data, training=is_training,
name=name, momentum=momentum,
epsilon=eps)
# dropout 处理
def deal_dropout(self, hidden_layer, keep_prob):
with tf.name_scope("dropout"):
tf.summary.scalar('dropout_keep_probability', keep_prob)
dropped = tf.nn.dropout(hidden_layer, keep_prob)
tf.summary.histogram('dropped', dropped)
return dropped
pass
# 神经网络层
def neural_layer(self, x, n_neuron, name, activation=None):
# 包含所有的计算节点对于这一层, name_scope 可写可不写
with tf.name_scope(name=name):
n_input = int(x.get_shape()[1])
stddev = 2 / np.sqrt(n_input)
# 这层里面的w可以看成是二维数组，每个神经元对于一组w参数
# truncated normal distribution 比 regular normal distribution的值小
# 不会出现任何大的权重值，确保慢慢的稳健的训练
# 使用这种标准方差会让收敛快
# w参数需要随机，不能为0，否则输出为0，最后调整都是一个幅度没意义
with tf.name_scope("weights"):
init_w = tf.truncated_normal((n_input, n_neuron), stddev=stddev)
w = tf.Variable(init_w, name="weight")
self.variable_summaries(w)
with tf.name_scope("biases"):
b = tf.Variable(tf.zeros([n_neuron]), name="bias")
self.variable_summaries(b)
with tf.name_scope("wx_plus_b"):
z = tf.matmul(x, w) + b
tf.summary.histogram('pre_activations', z)
if activation == "relu":
activation_result = tf.nn.relu(z)
tf.summary.histogram('activation_result', activation_result)
return activation_result
else:
return z
def dnn_layer(self, x, n_label, keep_prob):
# 隐藏层
with tf.name_scope("dnn"):
# 这里要注意矩阵匹配
x_scale = self.layer_bn(x, is_training=True, name="x_bn")
hidden_1 = self.neural_layer(x_scale, self.hidden_01, "hidden_01", activation="relu")
dropped_hidden_1 = self.deal_dropout(hidden_1, keep_prob)
hidden_scale_1 = self.layer_bn(dropped_hidden_1, is_training=True, name="hidden_bn_1")
hidden_2 = self.neural_layer(hidden_scale_1, self.hidden_02, "hidden_02", activation="relu")
dropped_hidden_2 = self.deal_dropout(hidden_2, keep_prob)
hidden_scale_2 = self.layer_bn(dropped_hidden_2, is_training=True, name="hidden_bn_2")
hidden_3 = self.neural_layer(hidden_scale_2, self.hidden_03, "hidden_03", activation="relu")
dropped_hidden_3 = self.deal_dropout(hidden_3, keep_prob)
hidden_scale_3 = self.layer_bn(dropped_hidden_3, is_training=True, name="hidden_bn_3")
logits = self.neural_layer(hidden_scale_3, n_label, name="logits")
return logits
# 定义Variable变量的数据汇总函数，我们计算出变量的mean、stddev、max、min
# 对这些标量数据使用tf.summary.scalar进行记录和汇总
# 使用tf.summary.histogram直接记录变量var的直方图数据
def variable_summaries(self, param):
with tf.name_scope('summaries'):
mean = tf.reduce_mean(param)
tf.summary.scalar('mean', mean)
with tf.name_scope('stddev'):
stddev = tf.sqrt(tf.reduce_mean(tf.square(param - mean)))
tf.summary.scalar('stddev', stddev)
tf.summary.scalar('max', tf.reduce_max(param))
tf.summary.scalar('min', tf.reduce_min(param))
tf.summary.histogram('histogram', param)
(7) The training Code mnist_train.py
#!/usr/bin/env python
# _*_ coding:utf-8 _*_
# ============================================
# @Time : 2020/02/05 13:52
# @Author : WanDaoYi
# @FileName : mnist_train.py
# ============================================
from datetime import datetime
import tensorflow as tf
from config import cfg
from core.common import Common
import numpy as np
class MnistTrain(object):
def __init__(self):
# 模型保存路径
self.model_save_path = cfg.TRAIN.MODEL_SAVE_PATH
self.log_path = cfg.LOG.LOG_SAVE_PATH
self.learning_rate = cfg.TRAIN.LEARNING_RATE
self.batch_size = cfg.TRAIN.BATCH_SIZE
self.n_epoch = cfg.TRAIN.N_EPOCH
self.common = Common()
# 读取数据和 维度
self.mnist_data, self.n_feature, self.n_label = self.common.read_data()
# 创建设计图
with tf.name_scope(name="input_data"):
self.x = tf.placeholder(dtype=tf.float32, shape=(None, self.n_feature), name="input_data")
self.y = tf.placeholder(dtype=tf.float32, shape=(None, self.n_label), name="input_labels")
with tf.name_scope(name="input_shape"):
# 784维度变形为图片保持到节点
# -1 代表进来的图片的数量、28，28是图片的高和宽，1是图片的颜色通道
image_shaped_input = tf.reshape(self.x, [-1, 28, 28, 1])
tf.summary.image('input', image_shaped_input, self.n_label)
self.keep_prob_dropout = cfg.TRAIN.KEEP_PROB_DROPOUT
self.keep_prob = tf.placeholder(tf.float32)
# 获取最后一层的输出
self.logits = self.common.dnn_layer(self.x, self.n_label, self.keep_prob_dropout)
# self.config = tf.ConfigProto()
# self.config.gpu_options.allow_growth = True
# self.sess = tf.Session(config=self.config)
self.sess = tf.InteractiveSession()
# 保存训练模型
self.saver = tf.train.Saver()
pass
# 灌入数据
def feed_dict(self, train_flag=True):
# 训练样本
if train_flag:
# 获取下一批次样本
x_data, y_data = self.mnist_data.train.next_batch(self.batch_size)
keep_prob = self.keep_prob_dropout
pass
# 验证样本
else:
x_data, y_data = self.mnist_data.test.images, self.mnist_data.test.labels
keep_prob = 1.0
pass
return {self.x: x_data, self.y: y_data, self.keep_prob: keep_prob}
pass
# 训练
def do_train(self):
# 计算loss 损失
with tf.name_scope("train_loss"):
# softmax_cross_entropy_with_logits 只会给 one-hot 编码
# sparse_softmax_cross_entropy_with_logits 只会给没有 one-hot 编码，使用的会给0-9分类号
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=self.y, logits=self.logits)
loss = tf.reduce_mean(cross_entropy, name="loss")
tf.summary.scalar("cross_entropy", loss)
pass
# 构建优化器
with tf.name_scope("optimizer"):
optimizer = tf.train.AdamOptimizer(learning_rate=self.learning_rate)
training_op = optimizer.minimize(loss=loss)
# 比对正确的率
# 只处理没 one-hot 编码，获取logits里面最大的那1位和y比较类别好是否相同，返回True或者False一组值
# correct = tf.nn.in_top_k(logits, y, 1) 处理 one-hot 编码后的 y
with tf.name_scope("accuracy"):
correct = tf.equal(tf.argmax(self.logits, 1), tf.argmax(self.y, 1))
acc = tf.reduce_mean(tf.cast(correct, tf.float32))
tf.summary.scalar("accuracy", acc)
# 因为我们之前定义了太多的tf.summary汇总操作，逐一执行这些操作太麻烦，
# 使用tf.summary.merge_all()直接获取所有汇总操作，以便后面执行
merged = tf.summary.merge_all()
# 定义两个tf.summary.FileWriter文件记录器再不同的子目录，分别用来存储训练和测试的日志数据
# 同时，将Session计算图sess.graph加入训练过程，这样再TensorBoard的GRAPHS窗口中就能展示
train_writer = tf.summary.FileWriter(self.log_path + 'train', self.sess.graph)
test_writer = tf.summary.FileWriter(self.log_path + 'test')
# 构建初始化变量
init_variable = tf.global_variables_initializer()
self.sess.run(init_variable)
test_acc = None
for epoch in range(self.n_epoch):
# 获取总样本数量
batch_number = self.mnist_data.train.num_examples
# 获取总样本一共几个批次
size_number = int(batch_number / self.batch_size)
for number in range(size_number):
summary, _ = self.sess.run([merged, training_op], feed_dict=self.feed_dict())
# 第几次循环
i = epoch * size_number + number + 1
train_writer.add_summary(summary, i)
pass
if number == size_number - 1:
# 获取下一批次样本
x_batch, y_batch = self.mnist_data.train.next_batch(self.batch_size)
acc_train = acc.eval(feed_dict={self.x: x_batch, self.y: y_batch})
print("acc_train: {}".format(acc_train))
# 测试
output = self.logits.eval(feed_dict={self.x: self.mnist_data.test.images})
y_perd = np.argmax(output, axis=1)
print("y_perd: {}".format(y_perd[: 5]))
y_true = np.argmax(self.mnist_data.test.labels, axis=1)
print("y_true: {}".format(y_true[: 5]))
# 验证 方法一
acc_test = acc.eval(feed_dict={self.x: self.mnist_data.test.images,
self.y: self.mnist_data.test.labels})
print("epoch: {}, acc_test: {}".format(epoch + 1, acc_test))
# 验证 方法二 两个方法，随便挑一个都可以的。
test_summary, acc_test_2 = self.sess.run([merged, acc], feed_dict=self.feed_dict(False))
print("epoch: {}, acc_test_2: {}".format(epoch + 1, acc_test_2))
test_writer.add_summary(test_summary, epoch + 1)
test_acc = acc_test
save_path = self.model_save_path + "acc={:.6f}".format(test_acc) + ".ckpt"
# 保存模型
self.saver.save(self.sess, save_path, global_step=self.n_epoch)
train_writer.close()
test_writer.close()
pass
if __name__ == "__main__":
# 代码开始时间
start_time = datetime.now()
print("开始时间: {}".format(start_time))
demo = MnistTrain()
demo.do_train()
# 代码结束时间
end_time = datetime.now()
print("结束时间: {}, 训练模型耗时: {}".format(end_time, end_time - start_time))
(8) Test Code mnist_test.py
#!/usr/bin/env python
# _*_ coding:utf-8 _*_
# ============================================
# @Time : 2020/02/05 13:52
# @Author : WanDaoYi
# @FileName : mnist_test.py
# ============================================
from datetime import datetime
import tensorflow as tf
import numpy as np
from config import cfg
from core.common import Common
class MnistTest(object):
def __init__(self):
self.common = Common()
# 读取数据和 维度
self.mnist_data, self.n_feature, self.n_label = self.common.read_data()
# ckpt 模型
self.test_ckpt_model = cfg.TEST.CKPT_MODEL_SAVE_PATH
print("test_ckpt_model: {}".format(self.test_ckpt_model))
# tf.reset_default_graph()
# 创建设计图
with tf.name_scope(name="input"):
self.x = tf.placeholder(dtype=tf.float32, shape=(None, self.n_feature), name="input_data")
self.y = tf.placeholder(dtype=tf.float32, shape=(None, self.n_label), name="input_labels")
# 获取最后一层的输出
self.logits = self.common.dnn_layer(self.x, self.n_label, 1)
# 使用 ckpt 模型测试
def do_ckpt_test(self):
saver = tf.train.Saver()
with tf.Session() as sess:
# 重载模型
saver.restore(sess, self.test_ckpt_model)
# 预测
output = self.logits.eval(feed_dict={self.x: self.mnist_data.test.images})
# 将 one-hot 预测值转为 数字
y_perd = np.argmax(output, axis=1)
print("预测值: {}".format(y_perd[: 5]))
# 真实值
y_true = np.argmax(self.mnist_data.test.labels, axis=1)
print("真实值: {}".format(y_true[: 5]))
pass
if __name__ == "__main__":
# 代码开始时间
start_time = datetime.now()
print("开始时间: {}".format(start_time))
demo = MnistTest()
# 使用 ckpt 模型测试
demo.do_ckpt_test()
# 代码结束时间
end_time = datetime.now()
print("结束时间: {}, 训练模型耗时: {}".format(end_time, end_time - start_time))
(9) The log operation
After training generates log, open a command window with cmd.
Enter the command: Path tensorboard --logdir = log folder --host = localhost
In tensorboard inside, we need to define --host = localhost localhost ip used to open the page, otherwise, only the copy machine show ip command window opens. Tensorboard port number is 6006, just "goog" Google word in turn. Here, tensorboard need to open a browser with Google, I tried Firefox and search dogs, they can not be opened.
I trained myself get tensorboard log open as follows:
tensorboard --logdir=G:\work_space\python_space\pro2018_space\wandao\mnist_pro\logs\mnist_log_train --host=localhost
The http: // localhost: 6006 Open in Google Chrome
In this project, I did not use early-stopping and fine-tuning, these two, have the opportunity then on. And, to the rear of the item, but also with other mAP accuracy. If there is interest in this project here to improve play, they can also get their first up.
Insufficient, please Gangster exhibitions.
Returns the neural network directory
Previous: depth articles - neural network (f) elaborate data enhancement and fine-tuning
