综述
前面俩小节已经讲了经典的alex-net和vgg网络，vgg-net在alex网络的基础上，测试了很多种加深网络的方式，得到了vgg16和vgg19最后的结果还不错，但是后来人们发现，在网络深度到达一定程度后，继续加深网络，会有俩个问题，一个是太远了，梯度消失，即数据分散在不再被激活的那个区域导致梯度为0消失了，这个可以通过norimalized核intermediate narmalization layers解决。二个是模型的准确率会迅速下滑，单并不是overfit造成的。作者提出了一个网络结构，通过重复利用输入来优化训练，结果出奇的好。
resnet网络结构
一般的，总体分为6层，第一层为一个卷积层加一个pool层，最后为一个全链接fc层，中间四层的每一层都由多个residual block组成，然后每个residual block又由2个或3个卷积层加shortcut connections(捷径，即加上了初始的输入x)组成，这样构成的深层次的卷积神经网络。
residual block
中间weight layer一般由2层或者3层卷积组成，层与层之间加上batch norimalization以及relu（何凯名再后续的文章中有提到怎么处理比较好，可以看这篇博客Binbin Xu），最后一层加上x后再relu，最为输出。
这里申明下，中间是4层，每层由多个residual block组成，每个block又由多个卷积层加上x（identity效果比较好，Binbin Xu identity好处很多，首先效果好，再者不会增加参数）
一个34层网络结构的例子
最右边即为一个32层的resnet网络，最上面一个卷积加池化，中间分别有3，4，6，3，这四层block，每个block由俩个卷积，即（3+4+6+3=16）×2=32，再加上最后一个fc层即34层的结构。
应用
CIFAR-10
发现复杂了可能效果还不好，所以就做了一个简单的模型，原始为32*32的图，padding了4位，然后再随机crop出32*32的图，接着便三个卷积层，分别为32*32×16，16*16×32，8*8×64，每层n个block，每个block俩个卷积层，再加上最后fc共6n+2层。说是110层效果最好，1000+层反而还不好了，可能是过拟合。
下面是部分实现代码，来自于ry/tensorflow-resnet
# This is what they use for CIFAR-10 and 100.
# See Section 4.2 in http://arxiv.org/abs/1512.03385
def inference_small(x,
is_training,
num_blocks=3, # 6n+2 total weight layers will be used.
use_bias=False, # defaults to using batch norm
num_classes=10):
c = Config()
c['is_training'] = tf.convert_to_tensor(is_training,
dtype='bool',
name='is_training')
c['use_bias'] = use_bias
c['fc_units_out'] = num_classes
c['num_blocks'] = num_blocks
c['num_classes'] = num_classes
inference_small_config(x, c)
def inference_small_config(x, c):
c['bottleneck'] = False
c['ksize'] = 3
c['stride'] = 1
with tf.variable_scope('scale1'):
c['conv_filters_out'] = 16
c['block_filters_internal'] = 16
c['stack_stride'] = 1
x = conv(x, c)
x = bn(x, c)
x = activation(x)
x = stack(x, c)
with tf.variable_scope('scale2'):
c['block_filters_internal'] = 32
c['stack_stride'] = 2
x = stack(x, c)
with tf.variable_scope('scale3'):
c['block_filters_internal'] = 64
c['stack_stride'] = 2
x = stack(x, c)
# post-net
x = tf.reduce_mean(x, reduction_indices=[1, 2], name="avg_pool")
if c['num_classes'] != None:
with tf.variable_scope('fc'):
x = fc(x, c)
return x
另一种部分实现代码，参考自wenxinxu/resnet-in-tensorflow
def inference(input_tensor_batch, n, reuse):
'''
The main function that defines the ResNet. total layers = 1 + 2n + 2n + 2n +1 = 6n + 2
:param input_tensor_batch: 4D tensor
:param n: num_residual_blocks
:param reuse: To build train graph, reuse=False. To build validation graph and share weights
with train graph, resue=True
:return: last layer in the network. Not softmax-ed
'''
layers = []
with tf.variable_scope('conv0', reuse=reuse):
conv0 = conv_bn_relu_layer(input_tensor_batch, [3, 3, 3, 16], 1)
activation_summary(conv0)
layers.append(conv0)
for i in range(n):
with tf.variable_scope('conv1_%d' %i, reuse=reuse):
if i == 0:
conv1 = residual_block(layers[-1], 16, first_block=True)
else:
conv1 = residual_block(layers[-1], 16)
activation_summary(conv1)
layers.append(conv1)
for i in range(n):
with tf.variable_scope('conv2_%d' %i, reuse=reuse):
conv2 = residual_block(layers[-1], 32)
activation_summary(conv2)
layers.append(conv2)
for i in range(n):
with tf.variable_scope('conv3_%d' %i, reuse=reuse):
conv3 = residual_block(layers[-1], 64)
layers.append(conv3)
assert conv3.get_shape().as_list()[1:] == [8, 8, 64]
with tf.variable_scope('fc', reuse=reuse):
in_channel = layers[-1].get_shape().as_list()[-1]
bn_layer = batch_normalization_layer(layers[-1], in_channel)
relu_layer = tf.nn.relu(bn_layer)
global_pool = tf.reduce_mean(relu_layer, [1, 2])
assert global_pool.get_shape().as_list()[-1:] == [64]
output = output_layer(global_pool, 10)
layers.append(output)
return layers[-1]
Imagenet
层差原文tabble1
了例如50层的话，中间四层分别（3+4+6+3）=16个block，每个block3个卷积，1×1，3×3，1×1，共16×3=48层，加上下俩层就五十层了。
下面是部分实现代码，来自于ry/tensorflow-resnet
def inference(x, is_training,
num_classes=1000,
num_blocks=[3, 4, 6, 3], # defaults to 50-layer network
use_bias=False, # defaults to using batch norm
bottleneck=True):
c = Config()
c['bottleneck'] = bottleneck
c['is_training'] = tf.convert_to_tensor(is_training,
dtype='bool',
name='is_training')
c['ksize'] = 3
c['stride'] = 1
c['use_bias'] = use_bias
c['fc_units_out'] = num_classes
c['num_blocks'] = num_blocks
c['stack_stride'] = 2
with tf.variable_scope('scale1'):
c['conv_filters_out'] = 64
c['ksize'] = 7
c['stride'] = 2
x = conv(x, c)
x = bn(x, c)
x = activation(x)
with tf.variable_scope('scale2'):
x = _max_pool(x, ksize=3, stride=2)
c['num_blocks'] = num_blocks[0]
c['stack_stride'] = 1
c['block_filters_internal'] = 64
x = stack(x, c)
with tf.variable_scope('scale3'):
c['num_blocks'] = num_blocks[1]
c['block_filters_internal'] = 128
assert c['stack_stride'] == 2
x = stack(x, c)
with tf.variable_scope('scale4'):
c['num_blocks'] = num_blocks[2]
c['block_filters_internal'] = 256
x = stack(x, c)
with tf.variable_scope('scale5'):
c['num_blocks'] = num_blocks[3]
c['block_filters_internal'] = 512
x = stack(x, c)
# post-net
x = tf.reduce_mean(x, reduction_indices=[1, 2], name="avg_pool")
if num_classes != None:
with tf.variable_scope('fc'):
x = fc(x, c)
return x