introduction
The object of post production and use tfrecord recorded dataset. (Step on numerous pits OTZ)
official tutorial here labeled a data read: Tensorflow import data and use the data
next example illustrates how to use tfrecord, suppose I want to be the task picture classification. First of all, I have a txt file that contains the paths of all images and their labels. There is a file that contains a lot of pictures of the folder. This is similar to the FIG:
After the data is ready, you can create and use TFrecored friends ~
Production TFrecord
Course, is to write a function to make TFrecord friends. Let's read txt file picture information to obtain a picture of the path of each and their labels, and then make some of this image pre-processing, and finally the picture and its corresponding label serialization, and create an image index and the label (ie, the following code "img_raw", "label"). Refer to code.
import random
import tensorflow as tf
from PIL import Image
def create_record(records_path, data_path, img_txt):
# 声明一个TFRecordWriter
writer = tf.python_io.TFRecordWriter(records_path)
# 读取图片信息，并且将读入的图片顺序打乱
img_list = []
with open(img_txt, 'r') as fr:
img_list = fr.readlines()
random.shuffle(img_list)
cnt = 0
# 遍历每一张图片信息
for img_info in img_list:
# 图片相对路径
img_name = img_info.split(' ')[0]
# 图片类别
img_cls = int(img_info.split(' ')[1])
img_path = data_path + img_name
img = Image.open(img_path)
# 对图片进行预处理（缩放，减去均值，二值化等等）
img = img.resize((128, 128))
img_raw = img.tobytes()
# 声明将要写入tfrecord的key值（即图片，标签）
example = tf.train.Example(
features=tf.train.Features(feature={
"label": tf.train.Feature(int64_list=tf.train.Int64List(value=[img_cls])),
'img_raw': tf.train.Feature(bytes_list=tf.train.BytesList(value=[img_raw]))
}))
# 将信息写入指定路径
writer.write(example.SerializeToString())
# 打印一些提示信息~
cnt += 1
if cnt % 1000 == 0:
print "processed %d images" % cnt
writer.close()
# 指定你想要生成tfrecord名称，图片文件夹路径，含有图片信息的txt文件
records_path = '/the/name/of/your/haha.tfrecords'
data_path = '/the/root/of/your/image_folder/'
img_txt = '/image/labels/list.txt'
create_record(records_path, data_path, img_txt)Use TFrecord
So far, the most convenient way is to use TensorFlow TFrecord the Dataset ApI . Here, we do advise do not read the data (cumbersome and outdated) a queue of ways.
First, we define a good _parse_function, this function is used to specify TFrecord in the index (ie, the above "img_raw", "label") . Then we define a TFRecordDataset, and to read the data by means of _parse_function. Finally, in order to get each round of training data, we just need to declare an extra iterator, each call get_next () can be friends.
# 定义如何解析TFrecord数据
def _parse_function(example_proto):
features = tf.parse_single_example(
example_proto,
features={
'label': tf.FixedLenFeature([], tf.int64),
'img_raw': tf.FixedLenFeature([], tf.string)
}
)
# 取出我们需要的数据（标签，图片）
label = features['label']
img = features['img_raw']
img = tf.decode_raw(img, tf.uint8)
# 对标签以及图片作预处理
img = tf.reshape(img, [128, 128, 3])
img = tf.cast(img, tf.float32) * (1. / 255) - 0.5
label = tf.cast(label, tf.int32)
return img, label
# 得到获取data batch的迭代器
def data_iterator(tfrecords):
# 声明TFRecordDataset
dataset = tf.contrib.data.TFRecordDataset(tfrecords)
dataset = dataset.map(_parse_function)
# 打乱顺序，无限重复训练数据，定义好batch size
dataset = dataset.shuffle(buffer_size=1000).repeat().batch(128)
# 定义one_shot_iterator。官方上有许多类型的iterrator，这种是最简单的
iterator = dataset.make_one_shot_iterator()
return iterator
# 指定TFrecords路径，得到training iterator。
train_tfrecords = '/your/path/to/haha.tfrecords'
train_iterator = data_iterator(train_tfrecords)
# 使用方式举例
with tf.Session(config= tfconfig) as sess:
tf.initialize_all_variables().run()
train_batch = train_iterator.get_next()
for step in xrange(50000):
train_x, train_y = sess.run(train_batch)
And then talk TensorFlow of Slim Module
This article ought to this end. But I still want to say TensorFlow really a bit difficult to use (I may be too weak haha). The main reason is its API too much, too fast update. However, we can quickly learn a lot of things (after all, it has a lot of supporters, which provides us with many examples and explain Bowen), such as the example of learning about Slim's .
Then talk to Slim this module, it is a new module in 2016 with the aim of reducing the amount of code to build the network. Personally feel really good use, highly recommended a try! ! ! (I do not believe you can go in and see URL above) good, paste the following piece of code shows use under slim as the end of Benpian it ~
slim = tf.contrib.slim
def MyNet(inputs, num_classes=7, is_training=True, keep_prob=0.5, scope='MyNet'):
net = tf.reshape(inputs, [-1, 128, 128, 3])
with slim.arg_scope([slim.conv2d, slim.fully_connected],
activation_fn=tf.nn.relu,
weights_regularizer=slim.l2_regularizer(0.0005)):
with slim.arg_scope([slim.conv2d],
stride=1,
padding='SAME',
weights_initializer=tf.contrib.layers.xavier_initializer_conv2d()):
net = slim.stack(net, slim.conv2d, [(8, [3, 3])], scope='conv1')
net = slim.max_pool2d(net, [2, 2], scope='pool1')
net = slim.stack(net, slim.conv2d, [(16, [3, 3]), (24, [3, 3])], scope='conv2')
net = slim.max_pool2d(net, [2, 2], scope='pool2')
net = slim.stack(net, slim.conv2d, [(24, [3, 3]), (24, [3, 3]), (36, [3, 3]), (36, [3, 3])], scope='conv3')
net = slim.flatten(net)
with slim.arg_scope([slim.fully_connected],
weights_initializer=tf.random_normal_initializer(stddev=0.01)):
net = slim.fully_connected(net, 2048, scope='fc6')
net = slim.dropout(net, keep_prob, scope='dropout6')
net = slim.fully_connected(net, 2048, scope='fc7')
net = slim.dropout(net, keep_prob, scope='dropout7')
net = slim.fully_connected(net, num_classes, activation_fn=None, scope='fc8')
return net
input_data = tf.placeholder(tf.float32, [None, 128, 128, 3])
output_logits = MyNet(input_data)