Implementing AutoEncoder on TensorFlow
I. Overview
AutoEncoder is roughly a learning method that compresses and reduces the dimension of the high-dimensional features of the data, and then goes through the reverse decoding process. In the learning process, the final result obtained by decoding is compared with the original data, and the loss function is reduced by modifying the weight bias parameter, and the ability to restore the original data is continuously improved. After the learning is completed, the results obtained in the first half of the encoding process can represent the low-dimensional "eigenvalues" of the original data. The learned autoencoder model can compress high-dimensional data to the desired dimension, and the principle is similar to PCA.
2. Model realization
1. AutoEncoder
First, on the MNIST dataset, feature compression and feature decompression are implemented, and the decompressed data is visually compared with the original data.
Look at the code first:
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
#导入MNIST数据
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=False)
learning_rate = 0.01
training_epochs = 10
batch_size = 256
display_step = 1
examples_to_show = 10
n_input = 784
# tf Graph input (only pictures)
X = tf.placeholder("float", [None, n_input])
#注意，如果解码器的最后一层使用了sigmoid()激活函数，那么X必须进行归一化处理。不然不可能训练出好的结果。
# 用字典的方式存储各隐藏层的参数
n_hidden_1 = 256 # 第一编码层神经元个数
n_hidden_2 = 128 # 第二编码层神经元个数
# 权重和偏置的变化在编码层和解码层顺序是相逆的
# 权重参数矩阵维度是每层的 输入*输出，偏置参数维度取决于输出层的单元数
weights = {
'encoder_h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])),
'encoder_h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
'decoder_h1': tf.Variable(tf.random_normal([n_hidden_2, n_hidden_1])),
'decoder_h2': tf.Variable(tf.random_normal([n_hidden_1, n_input])),
}
biases = {
'encoder_b1': tf.Variable(tf.random_normal([n_hidden_1])),
'encoder_b2': tf.Variable(tf.random_normal([n_hidden_2])),
'decoder_b1': tf.Variable(tf.random_normal([n_hidden_1])),
'decoder_b2': tf.Variable(tf.random_normal([n_input])),
}
# 每一层结构都是 xW + b
# 构建编码器
def encoder(x):
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights['encoder_h1']),
biases['encoder_b1']))
layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, weights['encoder_h2']),
biases['encoder_b2']))
return layer_2
# 构建解码器
def decoder(x):
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights['decoder_h1']),
biases['decoder_b1']))
layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, weights['decoder_h2']),
biases['decoder_b2']))
return layer_2
# 构建模型
encoder_op = encoder(X)
decoder_op = decoder(encoder_op)
# 预测
y_pred = decoder_op
y_true = X
# 定义代价函数和优化器
cost = tf.reduce_mean(tf.pow(y_true - y_pred, 2)) #最小二乘法
optimizer = tf.train.AdamOptimizer(learning_rate).minimize(cost)
with tf.Session() as sess:
# tf.initialize_all_variables() no long valid from
# 2017-03-02 if using tensorflow >= 0.12
if int((tf.__version__).split('.')[1]) < 12 and int((tf.__version__).split('.')[0]) < 1:
init = tf.initialize_all_variables()
else:
init = tf.global_variables_initializer()
sess.run(init)
# 首先计算总批数，保证每次循环训练集中的每个样本都参与训练，不同于批量训练
total_batch = int(mnist.train.num_examples/batch_size) #总批数
for epoch in range(training_epochs):
for i in range(total_batch):
batch_xs, batch_ys = mnist.train.next_batch(batch_size) # max(x) = 1, min(x) = 0
# Run optimization op (backprop) and cost op (to get loss value)
_, c = sess.run([optimizer, cost], feed_dict={X: batch_xs})
if epoch % display_step == 0:
print("Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(c))
print("Optimization Finished!")
encode_decode = sess.run(
y_pred, feed_dict={X: mnist.test.images[:examples_to_show]})
f, a = plt.subplots(2, 10, figsize=(10, 2))
for i in range(examples_to_show):
a[0][i].imshow(np.reshape(mnist.test.images[i], (28, 28)))
a[1][i].imshow(np.reshape(encode_decode[i], (28, 28)))
plt.show()Code interpretation:
First, import various libraries and datasets to be used, and define various parameters such as learning rate, number of training iterations, etc., which are clear and easy to modify later. Since the neural network structure of the autoencoder is very regular, it is the structure of xW + b, so the weight W of each layer and the variable tf.Variable of the bias b are placed in a dictionary, and the key value of the dictionary is passed. A clearer description. In terms of model construction, the encoder part and the decoder part are built separately, the activation function of each layer uses the sigmoid function, and the encoder usually uses the same activation function as the encoder. Usually the encoder part and the decoder part are a reciprocal process. For example, we design an encoder that reduces 784 dimensions to 256 dimensions and then to 128 dimensions. The decoder corresponds to decoding from 128 dimensions to 256 dimensions and then to 784 dimensions. dimension. Define the cost function. The cost function is expressed as the least squares expression between the output of the decoder and the original input. The optimizer uses the AdamOptimizer training phase to train all the training data in each cycle. After training, the training results are finally compared with the original data visualization, as shown in the figure below, with a high degree of reduction. If the number of training loops is increased or the number of layers of the autoencoder is increased, a better restoration effect can be obtained.
operation result:
2. Encoder
The working principle of the Encoder encoder is the same as that of the AutoEncoder. We visualize the low-dimensional "eigenvalues" obtained by encoding in the low-dimensional space to visually display the clustering effect of the data. Specifically, reduce the 784-dimensional MNIST data step by step from 784 to 128 to 64 to 10 and finally reduce it to 2 dimensions, and show it in a 2-dimensional coordinate system. The difference is that in the last layer of the encoder Instead of using a sigmoid activation function, we will use the default linear activation function so that the output is (-∞, +∞).
Full code:
import tensorflow as tf
import matplotlib.pyplot as plt
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=False)
learning_rate = 0.01
training_epochs = 10
batch_size = 256
display_step = 1
n_input = 784
X = tf.placeholder("float", [None, n_input])
n_hidden_1 = 128
n_hidden_2 = 64
n_hidden_3 = 10
n_hidden_4 = 2
weights = {
'encoder_h1': tf.Variable(tf.truncated_normal([n_input, n_hidden_1],)),
'encoder_h2': tf.Variable(tf.truncated_normal([n_hidden_1, n_hidden_2],)),
'encoder_h3': tf.Variable(tf.truncated_normal([n_hidden_2, n_hidden_3],)),
'encoder_h4': tf.Variable(tf.truncated_normal([n_hidden_3, n_hidden_4],)),
'decoder_h1': tf.Variable(tf.truncated_normal([n_hidden_4, n_hidden_3],)),
'decoder_h2': tf.Variable(tf.truncated_normal([n_hidden_3, n_hidden_2],)),
'decoder_h3': tf.Variable(tf.truncated_normal([n_hidden_2, n_hidden_1],)),
'decoder_h4': tf.Variable(tf.truncated_normal([n_hidden_1, n_input],)),
}
biases = {
'encoder_b1': tf.Variable(tf.random_normal([n_hidden_1])),
'encoder_b2': tf.Variable(tf.random_normal([n_hidden_2])),
'encoder_b3': tf.Variable(tf.random_normal([n_hidden_3])),
'encoder_b4': tf.Variable(tf.random_normal([n_hidden_4])),
'decoder_b1': tf.Variable(tf.random_normal([n_hidden_3])),
'decoder_b2': tf.Variable(tf.random_normal([n_hidden_2])),
'decoder_b3': tf.Variable(tf.random_normal([n_hidden_1])),
'decoder_b4': tf.Variable(tf.random_normal([n_input])),
}
def encoder(x):
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights['encoder_h1']),
biases['encoder_b1']))
layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, weights['encoder_h2']),
biases['encoder_b2']))
layer_3 = tf.nn.sigmoid(tf.add(tf.matmul(layer_2, weights['encoder_h3']),
biases['encoder_b3']))
# In order to facilitate the output of the encoding layer, the subsequent layer of the encoding layer does not use an activation function
layer_4 = tf.add(tf.matmul(layer_3, weights['encoder_h4']),
biases['encoder_b4'])
return layer_4
def decoder(x):
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights['decoder_h1']),
biases['decoder_b1']))
layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, weights['decoder_h2']),
biases['decoder_b2']))
layer_3 = tf.nn.sigmoid(tf.add(tf.matmul(layer_2, weights['decoder_h3']),
biases['decoder_b3']))
layer_4 = tf.nn.sigmoid(tf.add(tf.matmul(layer_3, weights['decoder_h4']),
biases['decoder_b4']))
return layer_4
encoder_op = encoder(X)
decoder_op = decoder(encoder_op)
y_pred = decoder_op
y_true = X
cost = tf.reduce_mean(tf.pow(y_true - y_pred, 2))
optimizer = tf.train.AdamOptimizer(learning_rate).minimize(cost)
with tf.Session() as sess:
# tf.initialize_all_variables() no long valid from
# 2017-03-02 if using tensorflow >= 0.12
if int((tf.__version__).split('.')[1]) < 12 and int((tf.__version__).split('.')[0]) < 1:
init = tf.initialize_all_variables()
else:
init = tf.global_variables_initializer()
sess.run(init)
total_batch = int(mnist.train.num_examples/batch_size)
for epoch in range(training_epochs):
for i in range(total_batch):
batch_xs, batch_ys = mnist.train.next_batch(batch_size) # max(x) = 1, min(x) = 0
_, c = sess.run([optimizer, cost], feed_dict={X: batch_xs})
if epoch % display_step == 0:
print("Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(c))
print("Optimization Finished!")
encoder_result = sess.run(encoder_op, feed_dict={X: mnist.test.images})
plt.scatter(encoder_result[:, 0], encoder_result[:, 1], c=mnist.test.labels)
plt.colorbar()
plt.show()
Experimental results:
It can be seen from the results that the 2-dimensional coding feature has a good clustering effect, each color in the figure represents a number, and the clustering is very good.
Of course, the results obtained in this experiment are just a brief introduction to AutoEncoder. To get the desired effect, a more complex autoencoder structure should be designed to obtain better distinguishing features.