CS231n-2017 Assignment3 RNN、LSTM、风格迁移

其他 2019-01-05 08:30:55 阅读次数: 0

一、RNN

所需完成的步骤记录在RNN_Captioning.ipynb文件中。

本例中所用的数据为Microsoft于2014年发布的COCO数据集。该数据集中和图像标注想拐的图片包含80000张训练图片和40000张验证图片。而这些图片的特征已通过VGG-16网络获得，存储在train2014_vgg16_fc7.h5和val2014_vgg16_fc7.h5文件中，每张图片由一个4096维的向量表征。为减少问题复杂度，本例还提供了经过PCA处理之后的特征，存储在train2014_vgg16_fc7_pca.h5和val2014_vgg16_fc7_pca.h5文件中，特征维度由4096维降低为512维。

图片和其标注示例如下，其中<START>和<END>为标注的起始和结束字符，<UNK>为词表中未出现的罕见词，另外为保证标注的长度一致，会在较短的标注后填充<NULL>特殊字符。

图 1. 图像标注示例
1. RNN的单步前向传播

前向传播的实现的方式，与上次作业大同小异，只不过这里将会实现循环网络层的逻辑。
考虑每次网络读入一个标注词时，将根据此次输入和此时的网络隐藏状态，计算新的网络隐藏状态。

rnn_layers.py文件中的rnn_step_forward()函数：

def rnn_step_forward(x, prev_h, Wx, Wh, b):
    
    next_h, cache = None, None
    # TODO: Implement a single forward step for the vanilla RNN. 
    next_h = tanh(np.dot(x, Wx) + np.dot(prev_h, Wh) + b)
    cache = (next_h, Wx, Wh, x, prev_h)
    return next_h, cache

其中tanh()为求取超正切值的辅助函数，一个考虑计算溢出异常的稳定版本如下：

def tanh(x):
    tmp = x.copy()
    tmp[tmp > 10] = 10
    tmp = np.exp(tmp*2)
    return (tmp - 1)/(tmp + 1)
2. RNN的单步反向传播

关于超正切函数的求导：

tanh ⁡ &ThinSpace; x = e x − e − x e x + e − x ⇒ tanh ⁡ ′ x = 1 − tanh ⁡ 2 x \tanh\, x = \frac{e^x - e^{-x}}{e^x + e^{-x}}\Rightarrow \tanh&#x27; x = 1 - \tanh^2 x
故在rnn_layers.py文件中实现rnn_step_backward()函数如下：

def rnn_step_backward(dnext_h, cache):
    dx, dprev_h, dWx, dWh, db = None, None, None, None, None
    # TODO: Implement the backward pass for a single step of a vanilla RNN. 
    next_h, Wx, Wh, x, prev_h = cache
    dtanh_h = dnext_h * (1 - next_h**2)
    dx = dtanh_h.dot(Wx.T)
    dprev_h = dtanh_h.dot(Wh.T)
    dWx = x.T.dot(dtanh_h)
    dWh = prev_h.T.dot(dtanh_h)
    db = np.sum(dtanh_h, axis=0)

    return dx, dprev_h, dWx, dWh, db
3. RNN的前向传播

网络读取一小批的标注数据x，(样本数为N，每条标注的长度为T)，并使用这批标注所对应图片的特征作为网络的初始隐藏状态h0，通过前向传播过程，获得各个样本在每一步推进中产生的隐藏状态h，并存储反向传播所需变量。
rnn_layers.py文件中的rnn_forward()函数：

def rnn_forward(x, h0, Wx, Wh, b):
    h, cache = None, None
    # TODO: Implement forward pass for a vanilla RNN running on a sequence of input data.
    N, T, D = x.shape
    _, H = h0.shape
    h = np.zeros((N, T, H))
    prev_h = h0
    for iter_time in range(T):
        h[:, iter_time, :],_ = rnn_step_forward(x[:, iter_time, :], prev_h, Wx, Wh, b)
        prev_h = h[:, iter_time, :]

    cache = (h0, h, Wx, Wh, x)

    return h, cache
4. RNN的反向传播

利用存储的变量实现反向传播过程。rnn_layers.py文件中的rnn_backward()函数：

def rnn_backward(dh, cache):

    dx, dh0, dWx, dWh, db = None, None, None, None, None
    # TODO: Implement the backward pass for a vanilla RNN running an entire sequence of data.
    N, T, H = dh.shape
    h0, h, Wx, Wh, x = cache
    dh0 = np.zeros_like(h0)
    dx = np.zeros_like(x)
    dWx = np.zeros_like(Wx)
    dWh = np.zeros_like(Wh)
    db = np.zeros(H)
    h = np.concatenate((h0[:, np.newaxis, :], h), axis=1)
    
    for iter_time in range(T):
        dnext_h = dh[:, -(iter_time+1), :] + dh0
        cache = (h[:, -(iter_time+1), :], Wx, Wh, x[:, -(iter_time+1), :], h[:, -(iter_time+2), :])
        dx_step, dh0, dWx_step, dWh_step, db_step = rnn_step_backward(dnext_h, cache)
        dx[:, -(iter_time+1), :] = dx_step
        dWx += dWx_step
        dWh += dWh_step
        db  += db_step

    return dx, dh0, dWx, dWh, db

注意其中梯度值的累积，这其实就是RNN共享参数的一种体现。

5. 字词的向量化表达

将图像标注中的词索引x转化为向量表达，并在后向传播时更新字词所对应的向量。
rnn_layers.py文件中的word_embedding_forward()函数：

def word_embedding_forward(x, W):
    out, cache = None, None
    # TODO: Implement the forward pass for word embeddings.
    out = W[x, :]
    cache = (x, W.shape)

    return out, cache

rnn_layers.py文件中的word_embedding_backward()函数：

def word_embedding_backward(dout, cache):
    dW = None
    # TODO: Implement the backward pass for word embeddings.
    x, shp = cache
    dW = np.zeros(shp)
    np.add.at(dW, x, dout)
    
    return dW
6. 考虑损失函数

rnn.py文件中的loss()函数：

def loss(self, features, captions):

        captions_in = captions[:, :-1]
        captions_out = captions[:, 1:]

        # You'll need this
        mask = (captions_out != self._null)

        # Weight and bias for the affine transform from image features to initial
        # hidden state
        W_proj, b_proj = self.params['W_proj'], self.params['b_proj']

        # Word embedding matrix
        W_embed = self.params['W_embed']

        # Input-to-hidden, hidden-to-hidden, and biases for the RNN
        Wx, Wh, b = self.params['Wx'], self.params['Wh'], self.params['b']

        # Weight and bias for the hidden-to-vocab transformation.
        W_vocab, b_vocab = self.params['W_vocab'], self.params['b_vocab']

        loss, grads = 0.0, {}
        ############################################################################
        # TODO: Implement the forward and backward passes for the CaptioningRNN.
        h0, cache_affine = affine_forward(features, W_proj, b_proj)  # (1)
        captions_in_vec, cache_embed = word_embedding_forward(captions_in, W_embed)  #(2)
        if self.cell_type == "rnn":
          h, cache_rnn = rnn_forward(captions_in_vec, h0, Wx, Wh, b)  # (3)
        elif self.cell_type == "lstm":
          h, cache_lstm = lstm_forward(captions_in_vec, h0, Wx, Wh, b)
        
        scores, cache_score = temporal_affine_forward(h, W_vocab, b_vocab)  # (4)
        loss, dscores = temporal_softmax_loss(scores, captions_out, mask)  # (5)

        dh, dW_vocab, db_vocab = temporal_affine_backward(dscores, cache_score) # (4)
        if self.cell_type == "rnn":
          dcaptions_in_vec, dh0, dWx, dWh, db = rnn_backward(dh, cache_rnn)  # (3)
        elif self.cell_type == "lstm":
          dcaptions_in_vec, dh0, dWx, dWh, db = lstm_backward(dh, cache_lstm)  # (3)
        
        dW_embed = word_embedding_backward(dcaptions_in_vec, cache_embed)  # (2)
        _, dW_proj, db_proj = affine_backward(dh0, cache_affine)  # (1)

        grads = {"W_vocab": dW_vocab, "b_vocab": db_vocab, 
                 "Wx": dWx, "Wh": dWh, "b": db,
                 "W_embed": dW_embed, "W_proj": dW_proj, "b_proj": db_proj}

        return loss, grads
7. 测试过程

rnn.py文件中的sample()函数：

def sample(self, features, max_length=30):
        N = features.shape[0]
        captions = self._null * np.ones((N, max_length), dtype=np.int32)

        # Unpack parameters
        W_proj, b_proj = self.params['W_proj'], self.params['b_proj']
        W_embed = self.params['W_embed']
        Wx, Wh, b = self.params['Wx'], self.params['Wh'], self.params['b']
        W_vocab, b_vocab = self.params['W_vocab'], self.params['b_vocab']

        # TODO: Implement test-time sampling for the model.
        c = np.zeros(b.shape[0]//4)
        h = features.dot(W_proj) + b_proj  # (1)
        captions[:, 0] = self._start


        for iter_time in range(1, max_length):
          prev_word = captions[:, iter_time-1]
          captions_in_vec, _ = word_embedding_forward(prev_word, W_embed)  #(2)
          if self.cell_type == "rnn":
            h, _ = rnn_step_forward(captions_in_vec, h, Wx, Wh, b)  # (3)
          else:
            h, c, _ = lstm_step_forward(captions_in_vec, h, c, Wx, Wh, b)  # (3)
          scores =  np.dot(h, W_vocab) + b_vocab  # (4)
          captions[:, iter_time] = np.argmax(scores, axis=1)
          
        pass

        return captions

二、LSTM

所需完成的步骤记录在LSTM_Captioning.ipynb文件中。

1. LSTM的单步前向传播

rnn_layers.py文件中的lstm_step_forward()函数：

def lstm_step_forward(x, prev_h, prev_c, Wx, Wh, b):

    next_h, next_c, cache = None, None, None
    # TODO: Implement the forward pass for a single timestep of an LSTM.
    H = b.shape[0]
    ifog = x.dot(Wx) + prev_h.dot(Wh) + b
    ifog = getIFOG(ifog, "T")
    next_c = getIFOG(ifog, 'f')*prev_c + getIFOG(ifog,'i')*getIFOG(ifog,"g")
    next_h = getIFOG(ifog, 'o')*tanh(next_c)
    cache = (x, prev_h, prev_c, Wx, Wh, next_c, ifog)

    return next_h, next_c, cache

其中getIFOG()函数为变换并拆分四门输出的辅助函数：

def getIFOG(ifog, which):
    H = ifog.shape[1]//4
    indx = {char:i*H for i, char in enumerate("ifog")}
    if which == "t" or which == "T":
        for char in indx:
            if char == "g":
                ifog[:, indx[char]:indx[char]+H] = tanh(ifog[:, indx[char]:indx[char]+H])
            else:
                ifog[:, indx[char]:indx[char]+H] = sigmoid(ifog[:, indx[char]:indx[char]+H])
        return ifog
    else:
        if which == "g":
            return ifog[:, indx[which]:indx[which]+H]
        else:
            return ifog[:, indx[which]:indx[which]+H]
2. LSTM的单步后向传播

rnn_layers.py文件中实现lstm_step_backward()函数：

def lstm_step_backward(dnext_h, dnext_c, cache):
    dx, dh, dc, dWx, dWh, db = None, None, None, None, None, None
    # TODO: Implement the backward pass for a single timestep of an LSTM.
    N, H = dnext_c.shape
    da = np.zeros((N, 4*H))

    x, prev_h, prev_c, Wx, Wh, next_c, ifog = cache

    tanhc_t = tanh(next_c)
    i = getIFOG(ifog, "i")
    f = getIFOG(ifog, "f")
    o = getIFOG(ifog, "o")
    g = getIFOG(ifog, "g")
    dh_c = dnext_h*o*(1-tanhc_t**2)
    setIFOG(da, "i", (dnext_c + dh_c)*g*(1-i)*i)
    setIFOG(da, "f", (dnext_c + dh_c)*prev_c*(1-f)*f)
    setIFOG(da, "o", dnext_h*tanhc_t*(1-o)*o)
    setIFOG(da, "g", (dnext_c + dh_c)*i*(1-g**2))

    dx = da.dot(Wx.T)
    dprev_h = da.dot(Wh.T)
    dprev_c = (dnext_c + dh_c) * f
    dWx = x.T.dot(da)
    dWh = prev_h.T.dot(da)
    db = np.sum(da, axis=0)
    
    return dx, dprev_h, dprev_c, dWx, dWh, db

由实现过程可见：LSTM中反馈到前一层的梯度除了dprev_h外，还包含dprev_c。其中dprev_h涉及与系数矩阵W的相乘，因此这一项在经历多步操作时，极易出现梯度爆炸或消失。而dprev_c这一项，只涉及元素相乘，因此，缓解了上述问题。

3. LSTM的前向传播

rnn_layers.py文件中的lstm_forward()函数：

def lstm_forward(x, h0, Wx, Wh, b):
    h, cache = None, None

    # TODO: Implement the forward pass for an LSTM over an entire timeseries.
    N, T, D = x.shape
    _, H = h0.shape
    h = np.zeros((N, T, H))
    c = np.zeros_like(h)
    prev_h = h0
    prev_c = np.zeros_like(prev_h)
    for iter_time in range(T):
        h[:, iter_time, :], c[:, iter_time, :], _ = lstm_step_forward(x[:, iter_time, :], prev_h, prev_c, Wx, Wh, b)
        prev_h = h[:, iter_time, :]
        prev_c = c[:, iter_time, :]

    cache = (h0, h, c, Wx, Wh, x, b)

    return h, cache
4. LSTM的反向传播

rnn_layers.py文件中的lstm_backward()函数：

def lstm_backward(dh, cache):
    dx, dh0, dWx, dWh, db = None, None, None, None, None

    # TODO: Implement the backward pass for an LSTM over an entire timeseries.
    h0, h, c, Wx, Wh, x, b = cache
    
    N, T, H = dh.shape
    dh0 = np.zeros_like(h0)
    dx = np.zeros_like(x)
    dWx = np.zeros_like(Wx)
    dWh = np.zeros_like(Wh)
    db = np.zeros(4*H)
    h = np.concatenate((h0[:, np.newaxis, :], h), axis=1)
    dnext_c = np.zeros_like(h0)
    c = np.concatenate((dnext_c[:, np.newaxis, :], c), axis=1)
    
    for iter_time in range(T):
        dnext_h = dh[:, -(iter_time+1), :] + dh0

        prev_h = h[:, -(iter_time+2), :]
        next_x = x[:, -(iter_time+1), :]
        ifog = next_x.dot(Wx) + prev_h.dot(Wh) + b
        ifog = getIFOG(ifog, "T")
        cache = (next_x, prev_h, c[:, -(iter_time+2), :], Wx, Wh, c[:, -(iter_time+1), :], ifog)
        dx_step, dh0, dnext_c, dWx_step, dWh_step, db_step = lstm_step_backward(dnext_h, dnext_c, cache)

        dx[:, -(iter_time+1), :] = dx_step
        dWx += dWx_step
        dWh += dWh_step
        db  += db_step

    return dx, dh0, dWx, dWh, db

三、网络的可视化

所需完成的步骤记录在NetworkVisualization-PyTorch.ipynb文件中(作业代码包里还提供了一个TensorFlow版本)。

在本部分中，将使用一个预训练的分类网络，来完成显著图显示、生成某类别的图像等任务。该预训练的网络是在ImageNet数据集上得到的SqueezeNet，其可得到与AlexNet相当的精度，但参数规模小到足以在CPU机器上完成本部分的工作。

1. 显著图

显著图(Saliency Map)展示了图像各部分对最终分类结果的影响程度。其计算方式是：网络终端的该图像的正确类别的得分对于图像像素的梯度。

NetworkVisualization-PyTorch.ipynb中的compute_saliency_maps函数：

def compute_saliency_maps(X, y, model):
    
    # Make sure the model is in "test" mode
    model.eval()
    
    # Wrap the input tensors in Variables
    X_var = Variable(X, requires_grad=True)
    y_var = Variable(y)
    saliency = None

    # TODO: Implement this function.

    y_pred = model(X_var)
    scores = y_pred.gather(1, y_var.view(-1, 1)).squeeze()
    loss = scores.sum()
    loss.backward()
    saliency = torch.max(torch.abs(X_var.grad), 1)[0]
    pass

    return saliency

在若干图片上的计算结果：

图 2. 显著图
2. 愚弄分类器

计算某图片在一个错误分类上的得分关于图片各像素的梯度，然后使用梯度上升法，修改图片，使得网络发生误判。
NetworkVisualization-PyTorch.ipynb中的make_fooling_image函数：

def make_fooling_image(X, target_y, model):
    
    # Initialize our fooling image to the input image, and wrap it in a Variable.
    X_fooling = X.clone()
    X_fooling_var = Variable(X_fooling, requires_grad=True)
    
    learning_rate = 1

    # TODO: Generate a fooling image X_fooling that the model will classify as the class target_y. 

    score = model(X_fooling_var)
    _, y_pred = torch.max(score,1)
    iter_count = 0
    while(y_pred.item() != target_y):
        iter_count += 1
        if iter_count%10 == 0:
            print(iter_count)
        loss = score[0, target_y]
        loss.backward()
        g = X_fooling_var.grad/X_fooling_var.grad.norm()
        X_fooling += learning_rate * g
        X_fooling_var.grad.zero_()
        model.zero_grad()
        score = model(X_fooling_var)
        _, y_pred = torch.max(score,1)
    print(iter_count)

    return X_fooling

最终生成的使网络误判的图片与原图的差异示例如下：

图 3. 使网络误判的图片与原图的差异
3. 生成最符合某类别的图片

优化目标函数定为某分类的得分再加上正则项；将图片初始为随机噪声，使用梯度上升法，生成最符合该类别的图片。

NetworkVisualization-PyTorch.ipynb中的create_class_visualization函数：

def create_class_visualization(target_y, model, dtype, **kwargs):

    model.type(dtype)
    l2_reg = kwargs.pop('l2_reg', 1e-3)
    learning_rate = kwargs.pop('learning_rate', 25)
    num_iterations = kwargs.pop('num_iterations', 100)
    blur_every = kwargs.pop('blur_every', 10)
    max_jitter = kwargs.pop('max_jitter', 16)
    show_every = kwargs.pop('show_every', 25)

    # Randomly initialize the image as a PyTorch Tensor, and also wrap it in
    # a PyTorch Variable.
    img = torch.randn(1, 3, 224, 224).mul_(1.0).type(dtype)
    img_var = Variable(img, requires_grad=True)

    for t in range(num_iterations):
        # Randomly jitter the image a bit; this gives slightly nicer results
        ox, oy = random.randint(0, max_jitter), random.randint(0, max_jitter)
        img.copy_(jitter(img, ox, oy))

        # TODO: Use the model to compute the gradient of the score for the     #
        # class target_y with respect to the pixels of the image, and make a   #
        # gradient step on the image using the learning rate.
        score = model(img_var)
        loss = score[0, target_y] - l2_reg * img.norm()**2
        loss.backward()
        img += learning_rate*img_var.grad
        img_var.grad.zero_()
        model.zero_grad()
        
        # Undo the random jitter
        img.copy_(jitter(img, -ox, -oy))

        # As regularizer, clamp and periodically blur the image
        for c in range(3):
            lo = float(-SQUEEZENET_MEAN[c] / SQUEEZENET_STD[c])
            hi = float((1.0 - SQUEEZENET_MEAN[c]) / SQUEEZENET_STD[c])
            img[:, c].clamp_(min=lo, max=hi)
        if t % blur_every == 0:
            blur_image(img, sigma=0.5)
        
        # Periodically show the image
        if t == 0 or (t + 1) % show_every == 0 or t == num_iterations - 1:
            plt.imshow(deprocess(img.clone().cpu()))
            class_name = class_names[target_y]
            plt.title('%s\nIteration %d / %d' % (class_name, t + 1, num_iterations))
            plt.gcf().set_size_inches(4, 4)
            plt.axis('off')
            plt.show()

    return deprocess(img.cpu())

对狼蛛类别的一个反演结果如下所示：

图 4. 狼蛛类的反演图像

四、风格迁移

风格迁移问题的损失函数由三部分构成：

  • 内容偏差：该部分由生成图片和源内容图片，在特征空间中的差异描述。

    StyleTransfer-PyTorch.ipynb中的content_loss()函数：

    def content_loss(content_weight, content_current, content_original):
    lc = content_weight * (content_current - content_original).norm()**2
    return lc

  • 风格偏差：该部分由生成图片和源风格图片的Gram矩阵之间的差异描述。

    StyleTransfer-PyTorch.ipynb中的gram_matrix()函数：

    def gram_matrix(features, normalize=True):

    N, C, H, W = features.shape
    gram = Variable(torch.zeros(N, C, C))
    x = features.view(N, C, -1)
    for i in range(N):
        gram[i] = torch.mm(x[i], x[i].t())
    if normalize:
        gram = gram/H/W/C
    return gram

    StyleTransfer-PyTorch.ipynb中的style_loss()函数：

    def style_loss(feats, style_layers, style_targets, style_weights):
    style_loss = Variable(torch.FloatTensor([0]))
    for i, indx in enumerate(style_layers):
        gram = gram_matrix(feats[indx])
        style_loss += style_weights[i] * torch.sum((gram - style_targets[i]).norm()**2)
    return style_loss

注意其中style_loss应初始化为torch.Variable，以记录与其相关的运算，用于反向传播时梯度的累积。

  • 图像变分惩罚项：该部分描述了图像的平滑度。

    StyleTransfer-PyTorch.ipynb中的tv_loss()函数：

    def tv_loss(img, tv_weight):
    x = torch.sum((img[:, :, 1:, :] - img[:, :, 0:-1, :]).norm()**2) + torch.sum((img[:, :, :, 1:] - img[:, :, :, 0:-1]).norm()**2)
    return tv_weight * xs

一个以梵高的星空为风格的迁移示例如下：

图 5. 内容图片与风格图片
图 6. 风格迁移的结果

猜你喜欢

转载自blog.csdn.net/suredied/article/details/83904653
今日推荐
面壁智能发布 Eurux-8x22B 开源大模型 —— 堪称「理科状元」
开源日报 | 谷歌扶持鸿蒙上位；开源Rabbit R1；Docker加持的安卓手机；微软的焦虑和野心；海尔电器把开放平台关了
中国码农的“35岁魔咒”
蘭雅 CorelDRAW 插件 2024.5.1 国际劳动节版，免费下载
Arc Browser for Windows 1.0 正式 GA
90后程序员开发视频搬运软件、不到一年获利超 700 万，结局很刑！
周排行
【转】spring中对控制反转和依赖注入的理解
tms webcore 安装和使用
java程序员进阶相关书籍
SpringMVC接受请求参数、
如何保存训练好的机器学习模型
MyEclipse、Eclipse设置项目JDK的三个地方
商超行业微信小程序开发定制一般多少钱 （行业技术人员解读）
Markdown编辑器语言——30分钟入门到到精通
Linux系统下MongoDB的简单安装与基本操作
Power Strings
每日归档
更多
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)
2024-04-28(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022