简介
介绍DeepDream的原理并用TensorFlow实现
效果
先来看一下DeepDream的效果，本来是这样一张图片
经过DeepDream处理之后就有可能变成这样
有点奇特和梦幻，也有点不明所以、精神污染
原理
大多时候我们是根据给定的数据和标签，去训练和调整网络的参数
不过也有时候，我们是固定网络的参数，根据某个损失函数调整输入数据，例如在图像风格迁移里，根据内容损失函数和风格损失函数调整合成的图片
对于常见的图片分类模型，输入一张图片，网络中的每个tensor会输出相应的响应值，值越大说明这个tensor越“喜欢”这张图片
比如输入一张狗的图片，网络中用于识别和分类狗的tensor就会输出较大的响应值
把优化目标设为最大化某个tensor的响应值，以此来调整输入图片，这就是DeepDream的原理
举例来说，为了满足一个喜欢狗的tensor，我们将原始图片中像狗的一些蛛丝马迹进行调整和放大，从而使得这一tensor的响应值更大
实现
加载库
# -*- coding: utf-8 -*-
import tensorflow as tf
import numpy as np
import cv2
from imageio import imread, imsave, mimsave
import matplotlib.pyplot as plt
%matplotlib inline
from scipy.ndimage.filters import gaussian_filter
复制代码加载图片分类模型，这里使用inception5h
layer_names = ['conv2d0', 'conv2d1', 'conv2d2',
'mixed3a', 'mixed3b', 'mixed4a', 'mixed4b', 'mixed4c', 'mixed4d', 'mixed4e',
'mixed5a', 'mixed5b']
graph = tf.Graph()
with graph.as_default():
with tf.gfile.FastGFile('inception5h.pb', 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
tf.import_graph_def(graph_def, name='')
X = graph.get_tensor_by_name('input:0')
layers = [graph.get_tensor_by_name(name + ':0') for name in layer_names]
all_layers_names = [tensor.name for tensor in tf.get_default_graph().as_graph_def().node]
print(all_layers_names)
sess = tf.Session(graph=graph)
复制代码定义获取梯度tensor的函数、对原始图片按块计算梯度的函数
def get_gradient(tensor):
with graph.as_default():
return tf.gradients(tf.reduce_mean(tf.square(tensor)), X)[0]
def get_tile_size(num_pixels, tile_size=400):
num_tiles = max(1, int(round(num_pixels / tile_size)))
return int(np.ceil(num_pixels / num_tiles))
def tiled_gradient(gradient, image, tile_size=400):
grad = np.zeros_like(image)
H, W, _ = image.shape
h = get_tile_size(H, tile_size)
h_4 = h // 4
w = get_tile_size(W, tile_size)
w_4 = w // 4
h_start = np.random.randint(-3 * h_4, -h_4)
while h_start < H:
h_end = h_start + h
h_start_lim = max(h_start, 0)
h_end_lim = min(h_end, H)
w_start = np.random.randint(-3 * w_4, -w_4)
while w_start < W:
w_end = w_start + w
w_start_lim = max(w_start, 0)
w_end_lim = min(w_end, W)
g = sess.run(gradient, feed_dict={X: [image[h_start_lim: h_end_lim, w_start_lim: w_end_lim, :]]})[0]
g /= (np.std(g) + 1e-8)
grad[h_start_lim: h_end_lim, w_start_lim: w_end_lim, :] = g
w_start = w_end
h_start = h_end
return grad
复制代码根据梯度调整输入图片，即DeepDream
def dream(layer_tensor, image, iteration=10, step=3.0, tile_size=400):
img = image.copy()
gradient = get_gradient(layer_tensor)
for i in range(iteration):
grad = tiled_gradient(gradient, img)
sigma = (i * 4.0) / iteration + 0.5
grad = gaussian_filter(grad, 0.5 * sigma) + gaussian_filter(grad, sigma) + gaussian_filter(grad, 2 * sigma)
scaled_step = step / (np.std(grad) + 1e-8)
img += grad * scaled_step
img = np.clip(img, 0, 255)
return img
复制代码将原始图片进行缩放，对多个尺度进行DeepDream处理并叠加
def recursive_dream(layer_tensor, image, repeat=3, scale=0.7, blend=0.2, iteration=10, step=3.0, tile_size=400):
if repeat > 0:
sigma = 0.5
img_blur = gaussian_filter(image, (sigma, sigma, 0.0))
h0 = img_blur.shape[0]
w0 = img_blur.shape[1]
h1 = int(scale * h0)
w1 = int(scale * w0)
img_downscaled = cv2.resize(img_blur, (w1, h1))
img_dream = recursive_dream(layer_tensor, img_downscaled, repeat - 1, scale, blend, iteration, step, tile_size)
img_upscaled = cv2.resize(img_dream, (w0, h0))
image = blend * image + (1.0 - blend) * img_upscaled
image = np.clip(image, 0, 255)
return dream(layer_tensor, image, iteration, step, tile_size)
复制代码读取一张图片
image = imread('mountain.jpg')
image = image.astype(np.float32)
复制代码分别以12个tensor的响应值作为优化目标，对原始图片进行处理
for i in range(len(layers)):
print(layer_names[i])
result = recursive_dream(layers[i], image)
plt.figure(figsize=(10, 15))
plt.imshow(result / 255.)
plt.show()
imsave('imgs/%s.jpg' % layer_names[i], result)
复制代码conv2d2的DeepDream结果
mixed3a的DeepDream结果
mixed4c的DeepDream结果
mixed5a的DeepDream结果
随着tensor所在的层数变深，DeepDream优化出来的图形也更加复杂
除了将某个tensor整个作为目标，也可以仅选择一个filter的响应值进行优化
例如选择mixed4c的某个filter，可以看到不同的filter偏好的图形是不一样的
for i in range(10):
print('Filter %d of mixed4c' % i)
result = recursive_dream(layers[7][:, :, :, i], image)
plt.figure(figsize=(10, 15))
plt.imshow(result / 255.)
plt.show()
imsave('imgs/mixed4c_filter_%d.jpg' % i, result)
复制代码mixed4c的filter0对应结果
mixed4c的filter8对应结果
当然，也可以对一张图片反复执行DeepDream，优化出来的图形会变得越来越明显
img = image.copy()
imgs = []
for i in range(20):
print('Iteration %d of mixed4c' % i)
img = recursive_dream(layers[7], img)
plt.figure(figsize=(10, 15))
plt.imshow(img / 255.)
plt.show()
imgs.append(img)
mimsave('imgs/mixed4c多轮迭代结果.gif', imgs, fps=5)
复制代码结果有点鬼畜，可能是mixed4c比较喜欢狗吧……