第二课 词向量
第二课学习目标
- 学习词向量的概念
- 用Skip-thought模型训练词向量
- 学习使用PyTorch dataset和dataloader
- 学习定义PyTorch模型
- 学习torch.nn中常见的Module
- Embedding
- 学习常见的PyTorch operations
- bmm
- logsigmoid
- 保存和读取PyTorch模型
第二课使用的训练数据可以从以下链接下载到。
链接:https://pan.baidu.com/s/1tFeK3mXuVXEy3EMarfeWvg 密码:v2z5
在这一份notebook中，我们会（尽可能）尝试复现论文Distributed Representations of Words and Phrases and their Compositionality中训练词向量的方法. 我们会实现Skip-gram模型，并且使用论文中noice contrastive sampling的目标函数。
这篇论文有很多模型实现的细节，这些细节对于词向量的好坏至关重要。我们虽然无法完全复现论文中的实验结果，主要是由于计算资源等各种细节原因，但是我们还是可以大致展示如何训练词向量。
以下是一些我们没有实现的细节
- subsampling：参考论文section 2.3，好像就是直接删去一部分频率很高的词
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.utils.data as tud
from torch.nn.parameter import Parameter
from collections import Counter
import numpy as np
import random
import math
import pandas as pd
import scipy
import sklearn
from sklearn.metrics.pairwise import cosine_similarity
# 看看有没有GPU
USE_CUDA = torch.cuda.is_available()
# 为了保证实验结果可以复现，我们经常会把各种random seed固定在某一个值
random.seed(53113)
np.random.seed(53113)
torch.manual_seed(53113)
if USE_CUDA:
torch.cuda.manual_seed(53113)
# 设定一些超参数
K = 100 # number of negative samples，负例采样的内容，对于小一点的数据集20-25够了，对于大一点的数据集2-5个(paper 说)
C = 3 # nearby words threshold
NUM_EPOCHS = 2 # The number of epochs of training，就训练两个epoch，每一个epoch应该是要把所有词都当成中心词过一遍的
MAX_VOCAB_SIZE = 30000 # the vocabulary size，一共就取这么多次，期中29999个常用词，剩下的1个位unk(不常用)，不常用词的频数是所有非常用词的频数和
BATCH_SIZE = 128 # the batch size
LEARNING_RATE = 0.2 # the initial learning rate
EMBEDDING_SIZE = 100 # 希望最终得到的每一个词都用一个100维的向量来表示
LOG_FILE = "word-embedding.log" # 不是很懂
# tokenize函数，把一篇文本转化成一个个单词
def word_tokenize(text):
return text.split()- 从文本文件中读取所有的文字，通过这些文本创建一个vocabulary
- 由于单词数量可能太大，我们只选取最常见的MAX_VOCAB_SIZE个单词
- 我们添加一个UNK单词表示所有不常见的单词
- 我们需要记录单词到index的mapping，以及index到单词的mapping，单词的count，单词的(normalized) frequency，以及单词总数。
# 数据集私信发你，我也是剽的
with open("text8.train.txt", "r") as fin:
text = fin.read()
# 分词后的文本
text = [w for w in word_tokenize(text.lower())]
# 选择频数最多的29999个词，留下一个为止给unk
vocab = dict(Counter(text).most_common(MAX_VOCAB_SIZE-1))
# unk表示所有非常用词的总和
vocab["<unk>"] = len(text) - np.sum(list(vocab.values()))
# 通过1-30000的下标，可以访问具体的词
idx_to_word = [word for word in vocab.keys()]
# 通过词，可以知道其下标(1-30000)
word_to_idx = {
word:i for i, word in enumerate(idx_to_word)}
# 30000个词的频数，用来进一步计算频率
word_counts = np.array([count for count in vocab.values()], dtype=np.float32)
# 频数
word_freqs = word_counts / np.sum(word_counts)
# 论文要求，这样效果更好
word_freqs = word_freqs ** (3./4.)
word_freqs = word_freqs / np.sum(word_freqs) # 用来做 negative sampling
# 验证一下对不对
VOCAB_SIZE = len(idx_to_word)
VOCAB_SIZE
# 可以看到，最后一个是6.17111e+05，表示所有非常用词的频数综合
print(word_counts)
# 康康频率
print(word_freqs)[9.58035e+05 5.36684e+05 3.75233e+05 ... 2.00000e+01 2.00000e+01
6.17111e+05]
[1.6231162e-02 1.0509998e-02 8.0359895e-03 ... 5.0128656e-06 5.0128656e-06
1.1670408e-02]
实现Dataloader
一个dataloader需要以下内容：
- 把所有text编码成数字，然后用subsampling预处理这些文字(这里没用实现，思路就是去除频率过高的词)。
- 保存vocabulary，单词count，normalized word frequency
- 每个iteration sample一个中心词
- 根据当前的中心词返回context单词
- 根据中心词sample一些negative单词
- 返回单词的counts
这里有一个好的tutorial介绍如何使用PyTorch dataloader.
为了使用dataloader，我们需要定义以下两个function:
__len__function需要返回整个数据集中有多少个item__get__根据给定的index返回一个item
有了dataloader之后，我们可以轻松随机打乱整个数据集，拿到一个batch的数据等等。
# 先要继承torch.utils.data.Dateset类，方便下面的dataloader来迭代
# 这个类要定义init、len和getitem
class WordEmbeddingDataset(tud.Dataset):
def __init__(self, text, word_to_idx, idx_to_word, word_freqs, word_counts):
''' text: a list of words, all text from the training dataset
word_to_idx: the dictionary from word to idx
idx_to_word: idx to word mapping
word_freq: the frequency of each word
word_counts: the word counts
'''
super(WordEmbeddingDataset, self).__init__()
self.text_encoded = [word_to_idx.get(t, VOCAB_SIZE-1) for t in text]
self.text_encoded = torch.Tensor(self.text_encoded).long()
self.word_to_idx = word_to_idx
self.idx_to_word = idx_to_word
self.word_freqs = torch.Tensor(word_freqs)
self.word_counts = torch.Tensor(word_counts)
self.lst = [i for i in range(len(self.text_encoded))]
def __len__(self):
''' 返回整个数据集（所有单词）的长度
'''
return len(self.text_encoded)
def __getitem__(self, idx):
''' 这个function返回以下数据用于训练
- 中心词
- 这个单词附近的(positive)单词
- 随机采样的K个单词作为negative sample
'''
# 返回单个中心词的下标
center_word = self.text_encoded[idx]
# 返回中心词周围的2*C个正例,但论文中应该是包括中心词一共2*C个的
pos_indices = list(range(idx-C, idx)) + list(range(idx+1, idx+C+1))
# idx+？可能越界，所以取余数
pos_indices = [i%len(self.text_encoded) for i in pos_indices]
# 和中心词一样，取正例的下标
pos_words = self.text_encoded[pos_indices]
# 负例采样，根据所有单词的频率，选取(K*正例数)/个负例，True代表有放回
# 也就是说对于一个中心词，会有2*C-1的周围词(正例),每一个正例，按照频率随机选取k个负例
# 个人认为：下面的代码有误，负采样的时候是可能取周围词和中心词的的。
# neg_words = torch.multinomial(self.word_freqs, K * pos_words.shape[0], True)
# 个人修改：每一次根据取到的pos_words(6个，在word_freqs中先将他们删去，再去取)，即:
for i in pos_words: # 删掉6个周围词
self.lst.remove(i)
self.lst.remove(center_word) # 删掉一个中心词
self.word_freqs = self.word_freqs[self.lst] # Tensor的切片，删除正例和中心词
neg_words = torch.multinomial(self.word_freqs,K*pos_words.shape[0],True)
return center_word, pos_words, neg_words 创建dataset和dataloader
dataset = WordEmbeddingDataset(text, word_to_idx, idx_to_word, word_freqs, word_counts)
# dataloader,按照自定义的dataset迭代取训练集
dataloader = tud.DataLoader(dataset, batch_size=BATCH_SIZE, shuffle=True)定义PyTorch模型
- init 定义网络
- forward 定义前向传播，之前的一课中只是输出y_pred,然后进一步计算loss，而这里直接输出loss
# 继承torch.nn.Module
class EmbeddingModel(nn.Module):
def __init__(self, vocab_size, embed_size):
''' 初始化输出和输出embedding
'''
super(EmbeddingModel, self).__init__()
# 一共有30000个词，也就是说，按照onehot词向量，每一个词由3万维的向量表示
self.vocab_size = vocab_size
# embed后，每一个词由100维的向量表示
self.embed_size = embed_size
# 输出embedding层，即周围词的矩阵
self.out_embed = nn.Embedding(self.vocab_size, self.embed_size, sparse=False)
# 初始化的一种方式，不清楚，用于下面的self.out_embed.weight.data.uniform_(-initrange, initrange)
initrange = 0.5 / self.embed_size
self.out_embed.weight.data.uniform_(-initrange, initrange)
# 输入embedding层，即中心词的矩阵
self.in_embed = nn.Embedding(self.vocab_size, self.embed_size, sparse=False)
self.in_embed.weight.data.uniform_(-initrange, initrange)
def forward(self, input_labels, pos_labels, neg_labels):
# 之前的dataloader已经定义了batch_size,会一次取batch_size个样本并进行整合后输入
'''
input_labels: 中心词, [batch_size]
pos_labels: 中心词周围 context window 出现过的单词 [batch_size ,(window_size * 2)]
neg_labels: 中心词周围没有出现过的单词，从 negative sampling 得到 [batch_size, (window_size * 2 * K)]
return: loss, [batch_size]
'''
batch_size = input_labels.size(0)
# input:(B,vocab_size)*(vacab_size,embed_size)
input_embedding = self.in_embed(input_labels) # B * embed_size
# output之pos:(B,c*2,vocab_size)*(vocab_size,embed_size)（B不参与计算，可能使用bmm进行计算的）
pos_embedding = self.out_embed(pos_labels) # B * (2*C) * embed_size
# output之neg:(B,c*2*K,vocab_size)*(vocab_size,embed_size)(B不参与)
neg_embedding = self.out_embed(neg_labels) # B * (2*C * K) * embed_size
# 根据paper定义loss function:
# (B , (2*C) ,embed_size) * (B ,embed_size),为了使B不参与，用unsqueeze给后者添加一维，用bmm计算，然后再删掉该维
log_pos = torch.bmm(pos_embedding, input_embedding.unsqueeze(2)).squeeze() # B * (2*C)
# (B , (2*C * K) , embed_size) * (B ,embed_size),为了使B不参与，用unsqueeze给后者添加一维，用bmm计算，然后再删掉该维
log_neg = torch.bmm(neg_embedding, -input_embedding.unsqueeze(2)).squeeze() # B * (2*C*K)
# 按照行，将每一行的log_(pos/neg)求和，因为每一行代表着一个样本。用logsigmoid不容易梯度消失
log_pos = F.logsigmoid(log_pos).sum(1)
log_neg = F.logsigmoid(log_neg).sum(1) # batch_size
loss = log_pos + log_neg
# log probability 越大越好，去反之后，得loss
return -loss
# 最终需要的，是input embedding矩阵
def input_embeddings(self):
return self.in_embed.weight.data.cpu().numpy()
定义一个模型以及把模型移动到GPU
model = EmbeddingModel(VOCAB_SIZE, EMBEDDING_SIZE)
# 雨我无瓜
if USE_CUDA:
model = model.cuda()下面是评估模型的代码，以及训练模型的代码
def evaluate(filename, embedding_weights):
if filename.endswith(".csv"):
data = pd.read_csv(filename, sep=",")
else:
data = pd.read_csv(filename, sep="\t")
human_similarity = []
model_similarity = []
for i in data.iloc[:, 0:2].index:
word1, word2 = data.iloc[i, 0], data.iloc[i, 1]
if word1 not in word_to_idx or word2 not in word_to_idx:
continue
else:
word1_idx, word2_idx = word_to_idx[word1], word_to_idx[word2]
word1_embed, word2_embed = embedding_weights[[word1_idx]], embedding_weights[[word2_idx]]
model_similarity.append(float(sklearn.metrics.pairwise.cosine_similarity(word1_embed, word2_embed)))
human_similarity.append(float(data.iloc[i, 2]))
return scipy.stats.spearmanr(human_similarity, model_similarity)# , model_similarity
def find_nearest(word):
index = word_to_idx[word]
embedding = embedding_weights[index]
cos_dis = np.array([scipy.spatial.distance.cosine(e, embedding) for e in embedding_weights])
return [idx_to_word[i] for i in cos_dis.argsort()[:10]]训练模型：
- 模型一般需要训练若干个epoch
- 每个epoch我们都把所有的数据分成若干个batch
- 把每个batch的输入和输出都包装成cuda tensor
- forward pass，通过输入的句子预测每个单词的下一个单词
- 用模型的预测和正确的下一个单词计算cross entropy loss
- 清空模型当前gradient
- backward pass
- 更新模型参数
- 每隔一定的iteration输出模型在当前iteration的loss，以及在验证数据集上做模型的评估
optimizer = torch.optim.SGD(model.parameters(), lr=LEARNING_RATE)
for e in range(NUM_EPOCHS):
for i, (input_labels, pos_labels, neg_labels) in enumerate(dataloader):
# TODO
input_labels = input_labels.long()
pos_labels = pos_labels.long()
neg_labels = neg_labels.long()
if USE_CUDA:
input_labels = input_labels.cuda()
pos_labels = pos_labels.cuda()
neg_labels = neg_labels.cuda()
optimizer.zero_grad()
loss = model(input_labels, pos_labels, neg_labels).mean()
loss.backward()
optimizer.step()
if i % 100 == 0:
with open(LOG_FILE, "a") as fout:
fout.write("epoch: {}, iter: {}, loss: {}\n".format(e, i, loss.item()))
print("epoch: {}, iter: {}, loss: {}".format(e, i, loss.item()))
if i % 2000 == 0:
embedding_weights = model.input_embeddings()
sim_simlex = evaluate("simlex-999.txt", embedding_weights)
sim_men = evaluate("men.txt", embedding_weights)
sim_353 = evaluate("wordsim353.csv", embedding_weights)
with open(LOG_FILE, "a") as fout:
print("epoch: {}, iteration: {}, simlex-999: {}, men: {}, sim353: {}, nearest to monster: {}\n".format(
e, i, sim_simlex, sim_men, sim_353, find_nearest("monster")))
fout.write("epoch: {}, iteration: {}, simlex-999: {}, men: {}, sim353: {}, nearest to monster: {}\n".format(
e, i, sim_simlex, sim_men, sim_353, find_nearest("monster")))
embedding_weights = model.input_embeddings()
np.save("embedding-{}".format(EMBEDDING_SIZE), embedding_weights)
torch.save(model.state_dict(), "embedding-{}.th".format(EMBEDDING_SIZE))s1
epoch: 0, iter: 0, loss: 420.04736328125
epoch: 0, iteration: 0, simlex-999: SpearmanrResult(correlation=0.002806243285464091, pvalue=0.9309107582703205), men: SpearmanrResult(correlation=-0.03578915454199749, pvalue=0.06854012381329619), sim353: SpearmanrResult(correlation=0.02468906830123471, pvalue=0.6609497549092586), nearest to monster: ['monster', 'communism', 'bosses', 'microprocessors', 'infectious', 'debussy', 'unesco', 'tantamount', 'offices', 'tischendorf']
epoch: 0, iter: 100, loss: 278.9967041015625
epoch: 0, iter: 200, loss: 248.71990966796875
epoch: 0, iter: 300, loss: 202.95816040039062
epoch: 0, iter: 400, loss: 157.04776000976562
epoch: 0, iter: 500, loss: 137.83531188964844
epoch: 0, iter: 600, loss: 121.03585815429688
epoch: 0, iter: 700, loss: 105.300537109375
epoch: 0, iter: 800, loss: 114.10055541992188
epoch: 0, iter: 900, loss: 104.72723388671875
epoch: 0, iter: 1000, loss: 99.03569030761719
epoch: 0, iter: 1100, loss: 95.2179946899414
epoch: 0, iter: 1200, loss: 84.12557983398438
epoch: 0, iter: 1300, loss: 88.07209777832031
epoch: 0, iter: 1400, loss: 70.44454193115234
epoch: 0, iter: 1500, loss: 79.83641052246094
epoch: 0, iter: 1600, loss: 81.7451171875
epoch: 0, iter: 1700, loss: 75.91305541992188
epoch: 0, iter: 1800, loss: 65.86140441894531
epoch: 0, iter: 1900, loss: 69.81714630126953
epoch: 0, iter: 2000, loss: 71.05166625976562
epoch: 0, iteration: 2000, simlex-999: SpearmanrResult(correlation=-0.011490367338787073, pvalue=0.7225847577400916), men: SpearmanrResult(correlation=0.05671509287050605, pvalue=0.0038790264864563434), sim353: SpearmanrResult(correlation=-0.07381419228558825, pvalue=0.18921537418718104), nearest to monster: ['monster', 'harm', 'steel', 'dean', 'kansas', 'surgery', 'regardless', 'capitalism', 'offers', 'hockey']
epoch: 0, iter: 2100, loss: 59.19840621948242
epoch: 0, iter: 2200, loss: 60.21418762207031
epoch: 0, iter: 2300, loss: 63.848148345947266
epoch: 0, iter: 2400, loss: 65.58479309082031
epoch: 0, iter: 2500, loss: 66.90382385253906
epoch: 0, iter: 2600, loss: 54.61847686767578
epoch: 0, iter: 2700, loss: 56.45966339111328
epoch: 0, iter: 2800, loss: 58.255210876464844
epoch: 0, iter: 2900, loss: 59.65287399291992
epoch: 0, iter: 3000, loss: 48.22801971435547
epoch: 0, iter: 3100, loss: 42.94969177246094
epoch: 0, iter: 3200, loss: 49.372528076171875
epoch: 0, iter: 3300, loss: 46.12495422363281
epoch: 0, iter: 3400, loss: 58.97121047973633
epoch: 0, iter: 3500, loss: 48.31055450439453
epoch: 0, iter: 3600, loss: 47.07227325439453
epoch: 0, iter: 3700, loss: 46.4068603515625
epoch: 0, iter: 3800, loss: 49.55707931518555
epoch: 0, iter: 3900, loss: 44.38733673095703
epoch: 0, iter: 4000, loss: 48.730342864990234
epoch: 0, iteration: 4000, simlex-999: SpearmanrResult(correlation=0.0190424235850696, pvalue=0.5562848091306694), men: SpearmanrResult(correlation=0.05404895260610133, pvalue=0.00592548586032086), sim353: SpearmanrResult(correlation=-0.039572591538143916, pvalue=0.4819454801463242), nearest to monster: ['monster', 'electrical', 'northeast', 'surgery', 'entity', 'certainly', 'tea', 'establishing', 'archbishop', 'aging']
epoch: 0, iter: 4100, loss: 57.70344161987305
epoch: 0, iter: 4200, loss: 47.464820861816406
epoch: 0, iter: 4300, loss: 47.08036804199219
epoch: 0, iter: 4400, loss: 46.652706146240234
epoch: 0, iter: 4500, loss: 40.824310302734375
epoch: 0, iter: 4600, loss: 40.62211227416992
epoch: 0, iter: 4700, loss: 50.84752655029297
epoch: 0, iter: 4800, loss: 41.230072021484375
epoch: 0, iter: 4900, loss: 53.74473571777344
epoch: 0, iter: 5000, loss: 42.35053253173828
epoch: 0, iter: 5100, loss: 38.363189697265625
epoch: 0, iter: 5200, loss: 42.772552490234375
epoch: 0, iter: 5300, loss: 44.914913177490234
epoch: 0, iter: 5400, loss: 38.4688720703125
epoch: 0, iter: 5500, loss: 41.0843391418457
epoch: 0, iter: 5600, loss: 35.04629898071289
epoch: 0, iter: 5700, loss: 35.49506759643555
epoch: 0, iter: 5800, loss: 36.009666442871094
epoch: 0, iter: 5900, loss: 40.56498718261719
epoch: 0, iter: 6000, loss: 45.853214263916016
epoch: 0, iteration: 6000, simlex-999: SpearmanrResult(correlation=0.04213372810279324, pvalue=0.19281410892481102), men: SpearmanrResult(correlation=0.06483263975087832, pvalue=0.0009600352172924885), sim353: SpearmanrResult(correlation=-0.015385630136134733, pvalue=0.7846219761829791), nearest to monster: ['monster', 'raw', 'romantic', 'oregon', 'protest', 'brunei', 'cartoon', 'offers', 'certainly', 'ill']
epoch: 0, iter: 6100, loss: 39.977508544921875
epoch: 0, iter: 6200, loss: 35.47979736328125
epoch: 0, iter: 6300, loss: 38.61311340332031
epoch: 0, iter: 6400, loss: 38.735679626464844
epoch: 0, iter: 6500, loss: 41.1725959777832
epoch: 0, iter: 6600, loss: 37.390037536621094
epoch: 0, iter: 6700, loss: 39.51911926269531
epoch: 0, iter: 6800, loss: 47.12213897705078
epoch: 0, iter: 6900, loss: 41.91630172729492
epoch: 0, iter: 7000, loss: 38.11504364013672
epoch: 0, iter: 7100, loss: 38.12763214111328
epoch: 0, iter: 7200, loss: 36.93813705444336
epoch: 0, iter: 7300, loss: 40.82877731323242
epoch: 0, iter: 7400, loss: 36.211429595947266
epoch: 0, iter: 7500, loss: 36.141693115234375
epoch: 0, iter: 7600, loss: 38.152610778808594
epoch: 0, iter: 7700, loss: 38.90789031982422
epoch: 0, iter: 7800, loss: 36.30712127685547
epoch: 0, iter: 7900, loss: 34.192440032958984
epoch: 0, iter: 8000, loss: 39.182212829589844
epoch: 0, iteration: 8000, simlex-999: SpearmanrResult(correlation=0.05506138271487322, pvalue=0.0886781241789579), men: SpearmanrResult(correlation=0.06796632118931804, pvalue=0.0005362832465382729), sim353: SpearmanrResult(correlation=-0.00727317983344893, pvalue=0.897207043425527), nearest to monster: ['monster', 'raw', 'romantic', 'strategic', 'offers', 'invited', 'signature', 'piano', 'protest', 'bills']
epoch: 0, iter: 8100, loss: 35.08313751220703
epoch: 0, iter: 8200, loss: 33.23561096191406
epoch: 0, iter: 8300, loss: 36.047096252441406
epoch: 0, iter: 8400, loss: 37.01750946044922
epoch: 0, iter: 8500, loss: 33.679561614990234
epoch: 0, iter: 8600, loss: 36.492515563964844
epoch: 0, iter: 8700, loss: 34.439537048339844
epoch: 0, iter: 8800, loss: 38.89817428588867
epoch: 0, iter: 8900, loss: 34.17725372314453
epoch: 0, iter: 9000, loss: 33.869651794433594
epoch: 0, iter: 9100, loss: 33.63176727294922
epoch: 0, iter: 9200, loss: 35.203460693359375
epoch: 0, iter: 9300, loss: 36.060142517089844
epoch: 0, iter: 9400, loss: 35.6544303894043
epoch: 0, iter: 9500, loss: 35.01182556152344
epoch: 0, iter: 9600, loss: 35.48432540893555
epoch: 0, iter: 9700, loss: 34.940696716308594
epoch: 0, iter: 9800, loss: 33.99235534667969
epoch: 0, iter: 9900, loss: 35.14078903198242
epoch: 0, iter: 10000, loss: 34.10219192504883
epoch: 0, iteration: 10000, simlex-999: SpearmanrResult(correlation=0.0714732189475033, pvalue=0.02703637716635098), men: SpearmanrResult(correlation=0.07013186360584196, pvalue=0.00035356323424747736), sim353: SpearmanrResult(correlation=-0.0013966072615024432, pvalue=0.9802088977698729), nearest to monster: ['monster', 'adoption', 'logo', 'particle', 'isle', 'remainder', 'profit', 'rank', 'execution', 'outer']
epoch: 0, iter: 10100, loss: 33.885284423828125
epoch: 0, iter: 10200, loss: 39.90406036376953
epoch: 0, iter: 10300, loss: 34.071014404296875
epoch: 0, iter: 10400, loss: 35.23554229736328
epoch: 0, iter: 10500, loss: 35.033878326416016
epoch: 0, iter: 10600, loss: 36.56634521484375
epoch: 0, iter: 10700, loss: 34.755027770996094
epoch: 0, iter: 10800, loss: 37.447967529296875
epoch: 0, iter: 10900, loss: 37.32883834838867
epoch: 0, iter: 11000, loss: 34.621700286865234
epoch: 0, iter: 11100, loss: 34.79033660888672
epoch: 0, iter: 11200, loss: 33.45790100097656
epoch: 0, iter: 11300, loss: 34.915672302246094
epoch: 0, iter: 11400, loss: 33.67906188964844
epoch: 0, iter: 11500, loss: 33.42378616333008
epoch: 0, iter: 11600, loss: 33.216270446777344
epoch: 0, iter: 11700, loss: 35.964393615722656
epoch: 0, iter: 11800, loss: 32.547569274902344
epoch: 0, iter: 11900, loss: 32.87192153930664
epoch: 0, iter: 12000, loss: 37.79120635986328
epoch: 0, iteration: 12000, simlex-999: SpearmanrResult(correlation=0.07427469122590927, pvalue=0.021568044209408773), men: SpearmanrResult(correlation=0.07554039135518772, pvalue=0.00011870202106880258), sim353: SpearmanrResult(correlation=0.003874488949244921, pvalue=0.9451327287240687), nearest to monster: ['monster', 'adoption', 'immediate', 'patent', 'sphere', 'execution', 'shell', 'nucleus', 'ghost', 'label']
epoch: 0, iter: 12100, loss: 33.59938430786133
epoch: 0, iter: 12200, loss: 32.594879150390625
epoch: 0, iter: 12300, loss: 32.42393493652344
epoch: 0, iter: 12400, loss: 32.8863410949707
epoch: 0, iter: 12500, loss: 39.303016662597656
epoch: 0, iter: 12600, loss: 33.103118896484375
epoch: 0, iter: 12700, loss: 36.31195068359375
epoch: 0, iter: 12800, loss: 33.8329963684082
epoch: 0, iter: 12900, loss: 32.499595642089844
epoch: 0, iter: 13000, loss: 33.224632263183594
epoch: 0, iter: 13100, loss: 33.931884765625
epoch: 0, iter: 13200, loss: 33.35892105102539
epoch: 0, iter: 13300, loss: 33.33966064453125
epoch: 0, iter: 13400, loss: 34.09075164794922
epoch: 0, iter: 13500, loss: 33.52397918701172
epoch: 0, iter: 13600, loss: 34.18444061279297
epoch: 0, iter: 13700, loss: 33.96720886230469
epoch: 0, iter: 13800, loss: 34.23271942138672
epoch: 0, iter: 13900, loss: 33.36094665527344
epoch: 0, iter: 14000, loss: 35.998287200927734
epoch: 0, iteration: 14000, simlex-999: SpearmanrResult(correlation=0.07498900438956249, pvalue=0.0203380930498303), men: SpearmanrResult(correlation=0.07885185599812983, pvalue=5.8687463983198815e-05), sim353: SpearmanrResult(correlation=0.019838726849964704, pvalue=0.7245257659604268), nearest to monster: ['monster', 'tale', 'patent', 'garden', 'outer', 'nucleus', 'logo', 'indians', 'fate', 'ghost']
epoch: 0, iter: 14100, loss: 32.86090087890625
epoch: 0, iter: 14200, loss: 32.27300262451172
epoch: 0, iter: 14300, loss: 32.97502136230469
epoch: 0, iter: 14400, loss: 33.18888473510742
epoch: 0, iter: 14500, loss: 33.709564208984375
epoch: 0, iter: 14600, loss: 33.725990295410156
epoch: 0, iter: 14700, loss: 34.124961853027344
epoch: 0, iter: 14800, loss: 34.69652557373047
epoch: 0, iter: 14900, loss: 36.399696350097656
epoch: 0, iter: 15000, loss: 32.656532287597656
epoch: 0, iter: 15100, loss: 33.403133392333984
epoch: 0, iter: 15200, loss: 32.11627960205078
epoch: 0, iter: 15300, loss: 32.489803314208984
epoch: 0, iter: 15400, loss: 32.96385192871094
epoch: 0, iter: 15500, loss: 33.85535430908203
epoch: 0, iter: 15600, loss: 33.443634033203125
epoch: 0, iter: 15700, loss: 32.89921569824219
epoch: 0, iter: 15800, loss: 31.661029815673828
epoch: 0, iter: 15900, loss: 32.627262115478516
epoch: 0, iter: 16000, loss: 32.10541534423828
epoch: 0, iteration: 16000, simlex-999: SpearmanrResult(correlation=0.0788889724045409, pvalue=0.014643454855412137), men: SpearmanrResult(correlation=0.08118046638145521, pvalue=3.517646407074078e-05), sim353: SpearmanrResult(correlation=0.03869824262332756, pvalue=0.49168668781560065), nearest to monster: ['monster', 'tale', 'patent', 'garden', 'logo', 'headquarters', 'floor', 'nucleus', 'hotel', 'outer']
epoch: 0, iter: 16100, loss: 32.40728759765625
epoch: 0, iter: 16200, loss: 32.153541564941406
epoch: 0, iter: 16300, loss: 32.54335021972656
epoch: 0, iter: 16400, loss: 33.81620788574219
epoch: 0, iter: 16500, loss: 33.6131591796875
epoch: 0, iter: 16600, loss: 33.012855529785156
epoch: 0, iter: 16700, loss: 32.9858512878418
epoch: 0, iter: 16800, loss: 32.339019775390625
epoch: 0, iter: 16900, loss: 33.2204475402832
epoch: 0, iter: 17000, loss: 32.71576690673828
epoch: 0, iter: 17100, loss: 33.55080795288086
epoch: 0, iter: 17200, loss: 32.817447662353516
epoch: 0, iter: 17300, loss: 34.78331756591797
epoch: 0, iter: 17400, loss: 32.013267517089844
epoch: 0, iter: 17500, loss: 32.31776428222656
epoch: 0, iter: 17600, loss: 32.81449508666992
epoch: 0, iter: 17700, loss: 32.663665771484375
epoch: 0, iter: 17800, loss: 32.64860534667969
epoch: 0, iter: 17900, loss: 32.25948715209961
epoch: 0, iter: 18000, loss: 33.899532318115234
epoch: 0, iteration: 18000, simlex-999: SpearmanrResult(correlation=0.08197570796707307, pvalue=0.01118359931439746), men: SpearmanrResult(correlation=0.08119437744439352, pvalue=3.5067625057299385e-05), sim353: SpearmanrResult(correlation=0.048031348197188906, pvalue=0.39330365782911914), nearest to monster: ['monster', 'tale', 'patent', 'household', 'dialogue', 'floor', 'sphere', 'mouse', 'fate', 'skin']
epoch: 0, iter: 18100, loss: 31.952678680419922
epoch: 0, iter: 18200, loss: 32.561737060546875
epoch: 0, iter: 18300, loss: 31.917354583740234
epoch: 0, iter: 18400, loss: 32.31993103027344
epoch: 0, iter: 18500, loss: 32.442169189453125
epoch: 0, iter: 18600, loss: 32.37964630126953
epoch: 0, iter: 18700, loss: 32.223846435546875
epoch: 0, iter: 18800, loss: 32.205589294433594
epoch: 0, iter: 18900, loss: 32.872222900390625
epoch: 0, iter: 19000, loss: 32.515403747558594
epoch: 0, iter: 19100, loss: 33.08296203613281
epoch: 0, iter: 19200, loss: 32.536170959472656
epoch: 0, iter: 19300, loss: 32.39844512939453
epoch: 0, iter: 19400, loss: 33.58967971801758
epoch: 0, iter: 19500, loss: 32.6730842590332
epoch: 0, iter: 19600, loss: 33.223388671875
epoch: 0, iter: 19700, loss: 32.08860397338867
epoch: 0, iter: 19800, loss: 31.78927993774414
epoch: 0, iter: 19900, loss: 31.92531967163086
epoch: 0, iter: 20000, loss: 32.14461898803711
epoch: 0, iteration: 20000, simlex-999: SpearmanrResult(correlation=0.08376406816249372, pvalue=0.00952959087521674), men: SpearmanrResult(correlation=0.08428805462978844, pvalue=1.7391127961421946e-05), sim353: SpearmanrResult(correlation=0.049551103193172526, pvalue=0.3784887673298559), nearest to monster: ['monster', 'patent', 'sword', 'household', 'dialogue', 'comprehensive', 'mouse', 'label', 'plain', 'tale']
epoch: 0, iter: 20100, loss: 32.788509368896484
epoch: 0, iter: 20200, loss: 31.743305206298828
epoch: 0, iter: 20300, loss: 32.98844909667969
epoch: 0, iter: 20400, loss: 32.939300537109375
epoch: 0, iter: 20500, loss: 32.22157669067383
epoch: 0, iter: 20600, loss: 32.10664367675781
epoch: 0, iter: 20700, loss: 32.317832946777344
epoch: 0, iter: 20800, loss: 32.22321701049805
epoch: 0, iter: 20900, loss: 32.078826904296875
epoch: 0, iter: 21000, loss: 32.00135803222656
epoch: 0, iter: 21100, loss: 32.2218017578125
epoch: 0, iter: 21200, loss: 32.36552047729492
epoch: 0, iter: 21300, loss: 32.28803253173828
epoch: 0, iter: 21400, loss: 32.49916076660156
epoch: 0, iter: 21500, loss: 31.330402374267578
epoch: 0, iter: 21600, loss: 32.153507232666016
epoch: 0, iter: 21700, loss: 32.27666473388672
epoch: 0, iter: 21800, loss: 31.28035545349121
epoch: 0, iter: 21900, loss: 31.78491973876953
epoch: 0, iter: 22000, loss: 32.09901428222656
epoch: 0, iteration: 22000, simlex-999: SpearmanrResult(correlation=0.0841933673566154, pvalue=0.009166514672039517), men: SpearmanrResult(correlation=0.08568243547516359, pvalue=1.2577781665179613e-05), sim353: SpearmanrResult(correlation=0.05233237611768227, pvalue=0.3522765894341572), nearest to monster: ['monster', 'sword', 'hero', 'ghost', 'patent', 'tale', 'comprehensive', 'plain', 'household', 'goddess']
epoch: 0, iter: 22100, loss: 32.55064392089844
epoch: 0, iter: 22200, loss: 32.269989013671875
epoch: 0, iter: 22300, loss: 31.861957550048828
epoch: 0, iter: 22400, loss: 35.57160949707031
epoch: 0, iter: 22500, loss: 31.28049087524414
epoch: 0, iter: 22600, loss: 32.447288513183594
epoch: 0, iter: 22700, loss: 31.807647705078125
epoch: 0, iter: 22800, loss: 31.540283203125
epoch: 0, iter: 22900, loss: 31.646018981933594
epoch: 0, iter: 23000, loss: 32.140228271484375
epoch: 0, iter: 23100, loss: 31.19212532043457
epoch: 0, iter: 23200, loss: 32.096595764160156
epoch: 0, iter: 23300, loss: 32.60624313354492
epoch: 0, iter: 23400, loss: 31.942745208740234
epoch: 0, iter: 23500, loss: 32.21788787841797
epoch: 0, iter: 23600, loss: 32.34299087524414
epoch: 0, iter: 23700, loss: 31.90642547607422
epoch: 0, iter: 23800, loss: 31.761348724365234
epoch: 0, iter: 23900, loss: 32.32670211791992
epoch: 0, iter: 24000, loss: 32.27470397949219
epoch: 0, iteration: 24000, simlex-999: SpearmanrResult(correlation=0.08547535475215376, pvalue=0.008154549896277891), men: SpearmanrResult(correlation=0.08635481027650124, pvalue=1.073940217602237e-05), sim353: SpearmanrResult(correlation=0.05715118428542604, pvalue=0.309644216967956), nearest to monster: ['monster', 'sword', 'hero', 'ghost', 'plain', 'household', 'situated', 'brand', 'torah', 'mouse']
epoch: 0, iter: 24100, loss: 31.711109161376953
epoch: 0, iter: 24200, loss: 31.729236602783203
epoch: 0, iter: 24300, loss: 31.751216888427734
epoch: 0, iter: 24400, loss: 31.54802131652832
epoch: 0, iter: 24500, loss: 31.819448471069336
epoch: 0, iter: 24600, loss: 31.87582778930664
epoch: 0, iter: 24700, loss: 32.44230651855469
epoch: 0, iter: 24800, loss: 32.13909149169922
epoch: 0, iter: 24900, loss: 31.6838321685791
epoch: 0, iter: 25000, loss: 32.01523208618164
epoch: 0, iter: 25100, loss: 31.727489471435547
epoch: 0, iter: 25200, loss: 32.378543853759766
epoch: 0, iter: 25300, loss: 32.155052185058594
epoch: 0, iter: 25400, loss: 32.30049514770508
epoch: 0, iter: 25500, loss: 32.10628128051758
epoch: 0, iter: 25600, loss: 32.01287841796875
epoch: 0, iter: 25700, loss: 32.22496032714844
epoch: 0, iter: 25800, loss: 32.15202331542969
epoch: 0, iter: 25900, loss: 32.43567657470703
epoch: 0, iter: 26000, loss: 31.745975494384766
epoch: 0, iteration: 26000, simlex-999: SpearmanrResult(correlation=0.08715629365703002, pvalue=0.0069793574982666565), men: SpearmanrResult(correlation=0.08749437789759629, pvalue=8.194697761171436e-06), sim353: SpearmanrResult(correlation=0.05971657549964074, pvalue=0.28839311254438554), nearest to monster: ['monster', 'sword', 'household', 'hero', 'tale', 'priest', 'label', 'plain', 'mouse', 'ghost']
epoch: 0, iter: 26100, loss: 32.09526824951172
epoch: 0, iter: 26200, loss: 31.927221298217773
epoch: 0, iter: 26300, loss: 31.239913940429688
epoch: 0, iter: 26400, loss: 31.676891326904297
epoch: 0, iter: 26500, loss: 31.83584213256836
epoch: 0, iter: 26600, loss: 32.34405517578125
epoch: 0, iter: 26700, loss: 31.836318969726562
epoch: 0, iter: 26800, loss: 31.805145263671875
epoch: 0, iter: 26900, loss: 31.517250061035156
epoch: 0, iter: 27000, loss: 32.060646057128906
epoch: 0, iter: 27100, loss: 31.427961349487305
epoch: 0, iter: 27200, loss: 32.71056365966797
epoch: 0, iter: 27300, loss: 32.101768493652344
epoch: 0, iter: 27400, loss: 31.706729888916016
epoch: 0, iter: 27500, loss: 31.794944763183594
epoch: 0, iter: 27600, loss: 31.043569564819336
epoch: 0, iter: 27700, loss: 31.815420150756836
epoch: 0, iter: 27800, loss: 31.480987548828125
epoch: 0, iter: 27900, loss: 32.0
epoch: 0, iter: 28000, loss: 31.647756576538086
epoch: 0, iteration: 28000, simlex-999: SpearmanrResult(correlation=0.0879225360679704, pvalue=0.006495932231970623), men: SpearmanrResult(correlation=0.08939464976521133, pvalue=5.181905435780726e-06), sim353: SpearmanrResult(correlation=0.06028361484068362, pvalue=0.28383120456458), nearest to monster: ['monster', 'sword', 'plain', 'mouse', 'tale', 'hero', 'brand', 'patent', 'tail', 'ghost']
epoch: 0, iter: 28100, loss: 31.732290267944336
epoch: 0, iter: 28200, loss: 31.56043243408203
epoch: 0, iter: 28300, loss: 32.17532730102539
epoch: 0, iter: 28400, loss: 32.34858322143555
epoch: 0, iter: 28500, loss: 31.321521759033203
epoch: 0, iter: 28600, loss: 31.24187469482422
epoch: 0, iter: 28700, loss: 31.808574676513672
epoch: 0, iter: 28800, loss: 31.126705169677734
epoch: 0, iter: 28900, loss: 32.27989959716797
epoch: 0, iter: 29000, loss: 31.510923385620117
epoch: 0, iter: 29100, loss: 32.18346405029297
epoch: 0, iter: 29200, loss: 31.19722557067871
epoch: 0, iter: 29300, loss: 31.348796844482422
epoch: 0, iter: 29400, loss: 32.03580856323242
epoch: 0, iter: 29500, loss: 31.560871124267578
epoch: 0, iter: 29600, loss: 32.1707763671875
epoch: 0, iter: 29700, loss: 31.646257400512695
epoch: 0, iter: 29800, loss: 32.089317321777344
epoch: 0, iter: 29900, loss: 31.21417808532715
epoch: 0, iter: 30000, loss: 31.596721649169922
epoch: 0, iteration: 30000, simlex-999: SpearmanrResult(correlation=0.0908617796694403, pvalue=0.00490825911563686), men: SpearmanrResult(correlation=0.09006953525508496, pvalue=4.393754176783815e-06), sim353: SpearmanrResult(correlation=0.06781615126644898, pvalue=0.22783225512951796), nearest to monster: ['monster', 'hero', 'sword', 'mouse', 'nickname', 'tale', 'plain', 'ghost', 'expedition', 'tube']
epoch: 0, iter: 30100, loss: 31.1719970703125
epoch: 0, iter: 30200, loss: 31.563777923583984
epoch: 0, iter: 30300, loss: 31.362476348876953
epoch: 0, iter: 30400, loss: 31.93914222717285
epoch: 0, iter: 30500, loss: 31.46084213256836
epoch: 0, iter: 30600, loss: 31.61031723022461
epoch: 0, iter: 30700, loss: 32.19886779785156
epoch: 0, iter: 30800, loss: 31.53145980834961
epoch: 0, iter: 30900, loss: 32.54494094848633
epoch: 0, iter: 31000, loss: 31.622350692749023
epoch: 0, iter: 31100, loss: 31.624622344970703
epoch: 0, iter: 31200, loss: 32.21925354003906
epoch: 0, iter: 31300, loss: 31.355022430419922
epoch: 0, iter: 31400, loss: 31.890806198120117
epoch: 0, iter: 31500, loss: 31.63449478149414
epoch: 0, iter: 31600, loss: 31.105436325073242
epoch: 0, iter: 31700, loss: 31.645238876342773
epoch: 0, iter: 31800, loss: 31.808307647705078
epoch: 0, iter: 31900, loss: 31.988243103027344
epoch: 0, iter: 32000, loss: 31.83148193359375
epoch: 0, iteration: 32000, simlex-999: SpearmanrResult(correlation=0.0910634737532756, pvalue=0.004813389051845152), men: SpearmanrResult(correlation=0.09222228979601282, pvalue=2.5756952028504964e-06), sim353: SpearmanrResult(correlation=0.07123238137344272, pvalue=0.20520429313982647), nearest to monster: ['monster', 'hero', 'nickname', 'sword', 'tale', 'plain', 'mouse', 'ghost', 'tail', 'tube']
epoch: 0, iter: 32100, loss: 31.311588287353516
epoch: 0, iter: 32200, loss: 31.257244110107422
epoch: 0, iter: 32300, loss: 31.649892807006836
epoch: 0, iter: 32400, loss: 31.635969161987305
epoch: 0, iter: 32500, loss: 31.34613037109375
epoch: 0, iter: 32600, loss: 31.666229248046875
epoch: 0, iter: 32700, loss: 31.63262176513672
epoch: 0, iter: 32800, loss: 31.727909088134766
epoch: 0, iter: 32900, loss: 32.014007568359375
epoch: 0, iter: 33000, loss: 31.64935302734375
epoch: 0, iter: 33100, loss: 31.75027084350586
epoch: 0, iter: 33200, loss: 30.913625717163086
epoch: 0, iter: 33300, loss: 32.485591888427734
epoch: 0, iter: 33400, loss: 30.946617126464844
epoch: 0, iter: 33500, loss: 31.906150817871094
epoch: 0, iter: 33600, loss: 31.456090927124023
epoch: 0, iter: 33700, loss: 31.70574188232422
epoch: 0, iter: 33800, loss: 31.611658096313477
epoch: 0, iter: 33900, loss: 31.901599884033203
epoch: 0, iter: 34000, loss: 30.904211044311523
epoch: 0, iteration: 34000, simlex-999: SpearmanrResult(correlation=0.09100297719676911, pvalue=0.004841669162232116), men: SpearmanrResult(correlation=0.09320434671047619, pvalue=2.0108242547194325e-06), sim353: SpearmanrResult(correlation=0.07643014870593084, pvalue=0.1739669852121724), nearest to monster: ['monster', 'nickname', 'tale', 'mouse', 'brand', 'hero', 'plain', 'partner', 'owner', 'cave']
epoch: 0, iter: 34100, loss: 30.875843048095703
epoch: 0, iter: 34200, loss: 31.53815460205078
epoch: 0, iter: 34300, loss: 31.3868465423584
epoch: 0, iter: 34400, loss: 31.618576049804688
epoch: 0, iter: 34500, loss: 31.38482666015625
epoch: 0, iter: 34600, loss: 31.517066955566406
epoch: 0, iter: 34700, loss: 31.297931671142578
epoch: 0, iter: 34800, loss: 31.131715774536133
epoch: 0, iter: 34900, loss: 31.34206199645996
epoch: 0, iter: 35000, loss: 31.198501586914062
epoch: 0, iter: 35100, loss: 31.92325782775879
epoch: 0, iter: 35200, loss: 31.495628356933594
epoch: 0, iter: 35300, loss: 31.19044303894043
epoch: 0, iter: 35400, loss: 31.896709442138672
epoch: 0, iter: 35500, loss: 31.638015747070312
epoch: 0, iter: 35600, loss: 31.722248077392578
epoch: 0, iter: 35700, loss: 31.750402450561523
epoch: 0, iter: 35800, loss: 31.107473373413086
epoch: 0, iter: 35900, loss: 31.830018997192383
epoch: 0, iter: 36000, loss: 31.638286590576172
epoch: 0, iteration: 36000, simlex-999: SpearmanrResult(correlation=0.09287122747290566, pvalue=0.004034242774511441), men: SpearmanrResult(correlation=0.09638454243178861, pvalue=8.867706115523595e-07), sim353: SpearmanrResult(correlation=0.08196414667104787, pvalue=0.14474986358858538), nearest to monster: ['monster', 'nickname', 'plain', 'sword', 'tail', 'mouse', 'brand', 'hero', 'tale', 'shell']
epoch: 0, iter: 36100, loss: 30.960386276245117
epoch: 0, iter: 36200, loss: 31.629940032958984
epoch: 0, iter: 36300, loss: 31.541032791137695
epoch: 0, iter: 36400, loss: 31.05801773071289
epoch: 0, iter: 36500, loss: 31.969802856445312
epoch: 0, iter: 36600, loss: 31.290489196777344
epoch: 0, iter: 36700, loss: 31.409465789794922
epoch: 0, iter: 36800, loss: 31.444076538085938
epoch: 0, iter: 36900, loss: 31.494474411010742
epoch: 0, iter: 37000, loss: 31.12554931640625
epoch: 0, iter: 37100, loss: 31.744049072265625
epoch: 0, iter: 37200, loss: 31.608917236328125
epoch: 0, iter: 37300, loss: 31.441722869873047
epoch: 0, iter: 37400, loss: 31.544227600097656
epoch: 0, iter: 37500, loss: 31.359806060791016
epoch: 0, iter: 37600, loss: 31.130847930908203
epoch: 0, iter: 37700, loss: 32.14916229248047
epoch: 0, iter: 37800, loss: 31.148212432861328
epoch: 0, iter: 37900, loss: 31.835248947143555
epoch: 0, iter: 38000, loss: 31.421974182128906
epoch: 0, iteration: 38000, simlex-999: SpearmanrResult(correlation=0.09401565185194706, pvalue=0.003602024110356835), men: SpearmanrResult(correlation=0.09723017395213002, pvalue=7.101718335843492e-07), sim353: SpearmanrResult(correlation=0.08744795260499, pvalue=0.1196457667795805), nearest to monster: ['monster', 'nickname', 'plain', 'sword', 'tail', 'hero', 'shell', 'brand', 'mouse', 'cave']
epoch: 0, iter: 38100, loss: 30.881635665893555
epoch: 0, iter: 38200, loss: 31.16852569580078
epoch: 0, iter: 38300, loss: 31.63935089111328
epoch: 0, iter: 38400, loss: 31.25921058654785
epoch: 0, iter: 38500, loss: 31.57360076904297
epoch: 0, iter: 38600, loss: 31.1456298828125
epoch: 0, iter: 38700, loss: 31.269453048706055
epoch: 0, iter: 38800, loss: 31.55490493774414
epoch: 0, iter: 38900, loss: 31.626995086669922
epoch: 0, iter: 39000, loss: 31.255146026611328
epoch: 0, iter: 39100, loss: 31.211166381835938
epoch: 0, iter: 39200, loss: 31.450740814208984
epoch: 0, iter: 39300, loss: 31.785430908203125
epoch: 0, iter: 39400, loss: 30.558988571166992
epoch: 0, iter: 39500, loss: 31.32469940185547
epoch: 0, iter: 39600, loss: 31.5628604888916
epoch: 0, iter: 39700, loss: 31.271900177001953
epoch: 0, iter: 39800, loss: 31.499229431152344
epoch: 0, iter: 39900, loss: 31.45954704284668
epoch: 0, iter: 40000, loss: 30.844253540039062
epoch: 0, iteration: 40000, simlex-999: SpearmanrResult(correlation=0.09659229076909075, pvalue=0.0027787052714036363), men: SpearmanrResult(correlation=0.09859835382112378, pvalue=4.938796863215718e-07), sim353: SpearmanrResult(correlation=0.09091941260502377, pvalue=0.10559925075777613), nearest to monster: ['monster', 'nickname', 'plain', 'hero', 'cave', 'sword', 'tail', 'owner', 'dialogue', 'mouse']
epoch: 0, iter: 40100, loss: 31.289958953857422
epoch: 0, iter: 40200, loss: 31.427631378173828
epoch: 0, iter: 40300, loss: 30.93175506591797
epoch: 0, iter: 40400, loss: 31.097423553466797
epoch: 0, iter: 40500, loss: 31.367881774902344
epoch: 0, iter: 40600, loss: 30.997957229614258
epoch: 0, iter: 40700, loss: 31.378498077392578
epoch: 0, iter: 40800, loss: 31.591278076171875
epoch: 0, iter: 40900, loss: 31.236934661865234
epoch: 0, iter: 41000, loss: 31.594310760498047
epoch: 0, iter: 41100, loss: 31.448932647705078
epoch: 0, iter: 41200, loss: 30.75921058654785
epoch: 0, iter: 41300, loss: 31.807411193847656
epoch: 0, iter: 41400, loss: 30.96005630493164
epoch: 0, iter: 41500, loss: 31.805885314941406
epoch: 0, iter: 41600, loss: 31.190258026123047
epoch: 0, iter: 41700, loss: 31.110252380371094
epoch: 0, iter: 41800, loss: 31.04319190979004
epoch: 0, iter: 41900, loss: 30.97702407836914
epoch: 0, iter: 42000, loss: 31.0760440826416
epoch: 0, iteration: 42000, simlex-999: SpearmanrResult(correlation=0.09849883639358918, pvalue=0.0022842954860782523), men: SpearmanrResult(correlation=0.09878607981201826, pvalue=4.696928075901211e-07), sim353: SpearmanrResult(correlation=0.09607407823044349, pvalue=0.08718116473821737), nearest to monster: ['monster', 'cave', 'plain', 'mouse', 'nickname', 'diamond', 'dialogue', 'partner', 'hero', 'signature']
epoch: 0, iter: 42100, loss: 31.264083862304688
epoch: 0, iter: 42200, loss: 31.662830352783203
epoch: 0, iter: 42300, loss: 30.693662643432617
epoch: 0, iter: 42400, loss: 31.405860900878906
epoch: 0, iter: 42500, loss: 30.938379287719727
epoch: 0, iter: 42600, loss: 31.08720588684082
epoch: 0, iter: 42700, loss: 31.5560302734375
epoch: 0, iter: 42800, loss: 31.49104881286621
epoch: 0, iter: 42900, loss: 31.75652503967285
epoch: 0, iter: 43000, loss: 31.436534881591797
epoch: 0, iter: 43100, loss: 31.30294418334961
epoch: 0, iter: 43200, loss: 30.177589416503906
epoch: 0, iter: 43300, loss: 31.117063522338867
epoch: 0, iter: 43400, loss: 30.985565185546875
epoch: 0, iter: 43500, loss: 30.83687973022461
epoch: 0, iter: 43600, loss: 31.235471725463867
epoch: 0, iter: 43700, loss: 31.702655792236328
epoch: 0, iter: 43800, loss: 31.2994441986084
epoch: 0, iter: 43900, loss: 30.892574310302734
epoch: 0, iter: 44000, loss: 31.143707275390625
epoch: 0, iteration: 44000, simlex-999: SpearmanrResult(correlation=0.1008180671120631, pvalue=0.001791807298558225), men: SpearmanrResult(correlation=0.09959304604435494, pvalue=3.781135368008198e-07), sim353: SpearmanrResult(correlation=0.10352488934150769, pvalue=0.06521224014872654), nearest to monster: ['monster', 'mouse', 'plain', 'nickname', 'cave', 'sword', 'boat', 'dialogue', 'partner', 'signature']
epoch: 0, iter: 44100, loss: 30.712566375732422
epoch: 0, iter: 44200, loss: 31.178279876708984
epoch: 0, iter: 44300, loss: 31.13910484313965
epoch: 0, iter: 44400, loss: 31.227054595947266
epoch: 0, iter: 44500, loss: 31.095703125
epoch: 0, iter: 44600, loss: 31.12378692626953
epoch: 0, iter: 44700, loss: 31.701740264892578
epoch: 0, iter: 44800, loss: 30.913339614868164
epoch: 0, iter: 44900, loss: 31.539695739746094
epoch: 0, iter: 45000, loss: 31.188980102539062
epoch: 0, iter: 45100, loss: 30.845016479492188
epoch: 0, iter: 45200, loss: 30.882841110229492
epoch: 0, iter: 45300, loss: 31.02661895751953
epoch: 0, iter: 45400, loss: 31.336511611938477
epoch: 0, iter: 45500, loss: 31.420623779296875
epoch: 0, iter: 45600, loss: 31.25517463684082
epoch: 0, iter: 45700, loss: 31.28260040283203
epoch: 0, iter: 45800, loss: 31.164663314819336
epoch: 0, iter: 45900, loss: 31.538354873657227
epoch: 0, iter: 46000, loss: 30.74416732788086
epoch: 0, iteration: 46000, simlex-999: SpearmanrResult(correlation=0.10155071688717769, pvalue=0.0016577893244519035), men: SpearmanrResult(correlation=0.10163250557010925, pvalue=2.1692539986465238e-07), sim353: SpearmanrResult(correlation=0.10604008852454751, pvalue=0.058912219453194116), nearest to monster: ['monster', 'plain', 'nickname', 'sword', 'parent', 'mouse', 'dialogue', 'blade', 'boat', 'cave']
epoch: 0, iter: 46100, loss: 31.790225982666016
epoch: 0, iter: 46200, loss: 31.57557487487793
epoch: 0, iter: 46300, loss: 31.457191467285156
epoch: 0, iter: 46400, loss: 31.26146697998047
epoch: 0, iter: 46500, loss: 31.269039154052734
epoch: 0, iter: 46600, loss: 31.480712890625
epoch: 0, iter: 46700, loss: 31.428335189819336
epoch: 0, iter: 46800, loss: 30.886512756347656
epoch: 0, iter: 46900, loss: 31.599918365478516
epoch: 0, iter: 47000, loss: 31.132366180419922
epoch: 0, iter: 47100, loss: 30.962696075439453
epoch: 0, iter: 47200, loss: 31.67426300048828
epoch: 0, iter: 47300, loss: 30.73318862915039
epoch: 0, iter: 47400, loss: 31.673181533813477
epoch: 0, iter: 47500, loss: 31.30075454711914
epoch: 0, iter: 47600, loss: 31.427719116210938
epoch: 0, iter: 47700, loss: 31.111129760742188
epoch: 0, iter: 47800, loss: 31.14962387084961
epoch: 0, iter: 47900, loss: 31.174724578857422
epoch: 0, iter: 48000, loss: 31.114784240722656
epoch: 0, iteration: 48000, simlex-999: SpearmanrResult(correlation=0.10335720444738267, pvalue=0.0013657280173897167), men: SpearmanrResult(correlation=0.10165177484469035, pvalue=2.157784847483254e-07), sim353: SpearmanrResult(correlation=0.11041761731040038, pvalue=0.049149368437484346), nearest to monster: ['monster', 'plain', 'nickname', 'parent', 'sword', 'blade', 'tail', 'dialogue', 'cave', 'boat']
epoch: 0, iter: 48100, loss: 31.263320922851562
epoch: 0, iter: 48200, loss: 31.267719268798828
epoch: 0, iter: 48300, loss: 31.259817123413086
epoch: 0, iter: 48400, loss: 30.922761917114258
epoch: 0, iter: 48500, loss: 31.48914909362793
epoch: 0, iter: 48600, loss: 31.45376205444336
epoch: 0, iter: 48700, loss: 30.948339462280273
epoch: 0, iter: 48800, loss: 30.842824935913086
epoch: 0, iter: 48900, loss: 30.931697845458984
epoch: 0, iter: 49000, loss: 31.468204498291016
epoch: 0, iter: 49100, loss: 31.04726791381836
epoch: 0, iter: 49200, loss: 31.148698806762695
epoch: 0, iter: 49300, loss: 31.295198440551758
epoch: 0, iter: 49400, loss: 31.415983200073242
epoch: 0, iter: 49500, loss: 31.53121566772461
epoch: 0, iter: 49600, loss: 30.391773223876953
epoch: 0, iter: 49700, loss: 31.365924835205078
epoch: 0, iter: 49800, loss: 30.920448303222656
epoch: 0, iter: 49900, loss: 30.881540298461914
epoch: 0, iter: 50000, loss: 31.272510528564453
epoch: 0, iteration: 50000, simlex-999: SpearmanrResult(correlation=0.10413335271622073, pvalue=0.0012554545146236879), men: SpearmanrResult(correlation=0.10361287469529604, pvalue=1.251734153196469e-07), sim353: SpearmanrResult(correlation=0.11252176428274015, pvalue=0.04496085066226139), nearest to monster: ['monster', 'parent', 'sword', 'nickname', 'boat', 'plain', 'tail', 'leg', 'mouse', 'blade']
epoch: 0, iter: 50100, loss: 31.378141403198242
epoch: 0, iter: 50200, loss: 30.816102981567383
epoch: 0, iter: 50300, loss: 30.845239639282227
epoch: 0, iter: 50400, loss: 30.991004943847656
epoch: 0, iter: 50500, loss: 30.891719818115234
epoch: 0, iter: 50600, loss: 31.482940673828125
epoch: 0, iter: 50700, loss: 31.31090545654297
epoch: 0, iter: 50800, loss: 31.34703826904297
epoch: 0, iter: 50900, loss: 31.271032333374023
epoch: 0, iter: 51000, loss: 31.262798309326172
epoch: 0, iter: 51100, loss: 31.295764923095703
epoch: 0, iter: 51200, loss: 31.204692840576172
epoch: 0, iter: 51300, loss: 31.768779754638672
epoch: 0, iter: 51400, loss: 30.988128662109375
epoch: 0, iter: 51500, loss: 31.494434356689453
epoch: 0, iter: 51600, loss: 31.034160614013672
epoch: 0, iter: 51700, loss: 31.57693099975586
epoch: 0, iter: 51800, loss: 31.073469161987305
epoch: 0, iter: 51900, loss: 30.947439193725586
epoch: 0, iter: 52000, loss: 31.44693374633789
epoch: 0, iteration: 52000, simlex-999: SpearmanrResult(correlation=0.10512161472015334, pvalue=0.0011269247727856127), men: SpearmanrResult(correlation=0.10482327567400625, pvalue=8.899831939238548e-08), sim353: SpearmanrResult(correlation=0.11557228293582009, pvalue=0.03942341315579966), nearest to monster: ['monster', 'parent', 'plain', 'nickname', 'tail', 'blade', 'leg', 'sword', 'mouse', 'signature']
epoch: 0, iter: 52100, loss: 31.460336685180664
epoch: 0, iter: 52200, loss: 31.668170928955078
epoch: 0, iter: 52300, loss: 30.479820251464844
epoch: 0, iter: 52400, loss: 31.74282455444336
epoch: 0, iter: 52500, loss: 30.803956985473633
epoch: 0, iter: 52600, loss: 30.81981658935547
epoch: 0, iter: 52700, loss: 31.19491958618164
epoch: 0, iter: 52800, loss: 31.08262825012207
epoch: 0, iter: 52900, loss: 31.718488693237305
epoch: 0, iter: 53000, loss: 30.93524932861328
epoch: 0, iter: 53100, loss: 31.013084411621094
epoch: 0, iter: 53200, loss: 30.91759490966797
epoch: 0, iter: 53300, loss: 31.814682006835938
epoch: 0, iter: 53400, loss: 30.962512969970703
epoch: 0, iter: 53500, loss: 30.939342498779297
epoch: 0, iter: 53600, loss: 31.67828369140625
epoch: 0, iter: 53700, loss: 31.302024841308594
epoch: 0, iter: 53800, loss: 30.955366134643555
epoch: 0, iter: 53900, loss: 31.510623931884766
epoch: 0, iter: 54000, loss: 30.676362991333008
epoch: 0, iteration: 54000, simlex-999: SpearmanrResult(correlation=0.10549625057039985, pvalue=0.0010814608115210568), men: SpearmanrResult(correlation=0.10721055714110006, pvalue=4.4913037691946484e-08), sim353: SpearmanrResult(correlation=0.11769079714384932, pvalue=0.03592648928681193), nearest to monster: ['monster', 'cave', 'plain', 'tail', 'nickname', 'leg', 'parent', 'blade', 'ghost', 'sword']
epoch: 0, iter: 54100, loss: 30.862953186035156
epoch: 0, iter: 54200, loss: 31.025741577148438
epoch: 0, iter: 54300, loss: 31.63794708251953
epoch: 0, iter: 54400, loss: 31.128381729125977
epoch: 0, iter: 54500, loss: 31.111047744750977
epoch: 0, iter: 54600, loss: 31.013673782348633
epoch: 0, iter: 54700, loss: 31.087692260742188
epoch: 0, iter: 54800, loss: 31.236892700195312
epoch: 0, iter: 54900, loss: 31.256492614746094
epoch: 0, iter: 55000, loss: 31.33741569519043
epoch: 0, iter: 55100, loss: 30.920692443847656
epoch: 0, iter: 55200, loss: 30.418161392211914
epoch: 0, iter: 55300, loss: 31.029638290405273
epoch: 0, iter: 55400, loss: 30.816083908081055
epoch: 0, iter: 55500, loss: 31.28571891784668
epoch: 0, iter: 55600, loss: 30.913631439208984
epoch: 0, iter: 55700, loss: 31.216411590576172
epoch: 0, iter: 55800, loss: 30.906490325927734
epoch: 0, iter: 55900, loss: 31.138019561767578
epoch: 0, iter: 56000, loss: 31.294254302978516
epoch: 0, iteration: 56000, simlex-999: SpearmanrResult(correlation=0.10747042881452093, pvalue=0.000868628570627656), men: SpearmanrResult(correlation=0.10889856420740306, pvalue=2.7445360530151176e-08), sim353: SpearmanrResult(correlation=0.12059633732769914, pvalue=0.03156119729084597), nearest to monster: ['monster', 'leg', 'tail', 'plain', 'nickname', 'sword', 'cave', 'parent', 'signature', 'blade']
epoch: 0, iter: 56100, loss: 31.756465911865234
epoch: 0, iter: 56200, loss: 30.841522216796875
epoch: 0, iter: 56300, loss: 31.101408004760742
epoch: 0, iter: 56400, loss: 30.875770568847656
epoch: 0, iter: 56500, loss: 31.620193481445312
epoch: 0, iter: 56600, loss: 31.299055099487305
epoch: 0, iter: 56700, loss: 31.310087203979492
epoch: 0, iter: 56800, loss: 31.34184455871582
epoch: 0, iter: 56900, loss: 31.240936279296875
epoch: 0, iter: 57000, loss: 30.90418815612793
epoch: 0, iter: 57100, loss: 31.257062911987305
epoch: 0, iter: 57200, loss: 31.695873260498047
epoch: 0, iter: 57300, loss: 30.94247055053711
epoch: 0, iter: 57400, loss: 30.684371948242188
epoch: 0, iter: 57500, loss: 31.504837036132812
epoch: 0, iter: 57600, loss: 31.262527465820312
epoch: 0, iter: 57700, loss: 31.38228988647461
epoch: 0, iter: 57800, loss: 31.27547836303711
epoch: 0, iter: 57900, loss: 30.56299591064453
epoch: 0, iter: 58000, loss: 30.96505355834961
epoch: 0, iteration: 58000, simlex-999: SpearmanrResult(correlation=0.10726587013548153, pvalue=0.0008887284442590619), men: SpearmanrResult(correlation=0.1093378019236274, pvalue=2.411455598304873e-08), sim353: SpearmanrResult(correlation=0.12085774638225694, pvalue=0.031191656645013426), nearest to monster: ['monster', 'tail', 'mouse', 'plain', 'blade', 'cave', 'signature', 'dubbed', 'angel', 'leg']
epoch: 0, iter: 58100, loss: 30.774898529052734
epoch: 0, iter: 58200, loss: 31.404376983642578
epoch: 0, iter: 58300, loss: 31.18744659423828
epoch: 0, iter: 58400, loss: 30.777774810791016
epoch: 0, iter: 58500, loss: 31.30267333984375
epoch: 0, iter: 58600, loss: 31.17416763305664
epoch: 0, iter: 58700, loss: 30.96724510192871
epoch: 0, iter: 58800, loss: 31.406417846679688
epoch: 0, iter: 58900, loss: 30.588558197021484
epoch: 0, iter: 59000, loss: 30.476577758789062
epoch: 0, iter: 59100, loss: 30.78055763244629
epoch: 0, iter: 59200, loss: 31.018653869628906
epoch: 0, iter: 59300, loss: 31.332136154174805
epoch: 0, iter: 59400, loss: 31.59677505493164
epoch: 0, iter: 59500, loss: 31.528217315673828
epoch: 0, iter: 59600, loss: 30.44378662109375
epoch: 0, iter: 59700, loss: 30.718303680419922
epoch: 0, iter: 59800, loss: 30.775535583496094
epoch: 0, iter: 59900, loss: 31.164199829101562
epoch: 0, iter: 60000, loss: 31.144628524780273
epoch: 0, iteration: 60000, simlex-999: SpearmanrResult(correlation=0.10787079798749166, pvalue=0.0008305009335772662), men: SpearmanrResult(correlation=0.11117447271855486, pvalue=1.3962055373097627e-08), sim353: SpearmanrResult(correlation=0.12349889211192522, pvalue=0.027660751055994845), nearest to monster: ['monster', 'tail', 'signature', 'blade', 'angel', 'dubbed', 'mouse', 'pole', 'owner', 'plain']
epoch: 0, iter: 60100, loss: 31.20758056640625
epoch: 0, iter: 60200, loss: 30.711570739746094
epoch: 0, iter: 60300, loss: 30.836360931396484
epoch: 0, iter: 60400, loss: 30.35114097595215
epoch: 0, iter: 60500, loss: 30.544240951538086
epoch: 0, iter: 60600, loss: 31.264543533325195
epoch: 0, iter: 60700, loss: 31.218517303466797
epoch: 0, iter: 60800, loss: 31.23360824584961
epoch: 0, iter: 60900, loss: 30.85096549987793
epoch: 0, iter: 61000, loss: 30.768386840820312
epoch: 0, iter: 61100, loss: 31.50748634338379
epoch: 0, iter: 61200, loss: 30.46345329284668
epoch: 0, iter: 61300, loss: 30.543607711791992
epoch: 0, iter: 61400, loss: 30.628982543945312
epoch: 0, iter: 61500, loss: 31.45627784729004
epoch: 0, iter: 61600, loss: 31.070459365844727
epoch: 0, iter: 61700, loss: 30.569217681884766
epoch: 0, iter: 61800, loss: 30.83639907836914
epoch: 0, iter: 61900, loss: 31.005922317504883
epoch: 0, iter: 62000, loss: 31.41488265991211
epoch: 0, iteration: 62000, simlex-999: SpearmanrResult(correlation=0.11119875283206068, pvalue=0.0005685786512505508), men: SpearmanrResult(correlation=0.11318488733549789, pvalue=7.599257092187759e-09), sim353: SpearmanrResult(correlation=0.12779805415765372, pvalue=0.022646548827240445), nearest to monster: ['monster', 'tail', 'blade', 'signature', 'plain', 'mouse', 'pole', 'boat', 'owner', 'leg']
epoch: 0, iter: 62100, loss: 31.053007125854492
epoch: 0, iter: 62200, loss: 30.765029907226562
epoch: 0, iter: 62300, loss: 31.114418029785156
epoch: 0, iter: 62400, loss: 30.98143768310547
epoch: 0, iter: 62500, loss: 31.071922302246094
epoch: 0, iter: 62600, loss: 31.17368507385254
epoch: 0, iter: 62700, loss: 31.177242279052734
epoch: 0, iter: 62800, loss: 31.408926010131836
epoch: 0, iter: 62900, loss: 30.88961410522461
epoch: 0, iter: 63000, loss: 30.848337173461914
epoch: 0, iter: 63100, loss: 30.798885345458984
epoch: 0, iter: 63200, loss: 30.96042251586914
epoch: 0, iter: 63300, loss: 30.656030654907227
epoch: 0, iter: 63400, loss: 31.166887283325195
epoch: 0, iter: 63500, loss: 30.926340103149414
epoch: 0, iter: 63600, loss: 31.11106300354004
epoch: 0, iter: 63700, loss: 31.001605987548828
epoch: 0, iter: 63800, loss: 30.872831344604492
epoch: 0, iter: 63900, loss: 31.2712345123291
epoch: 0, iter: 64000, loss: 31.084636688232422
epoch: 0, iteration: 64000, simlex-999: SpearmanrResult(correlation=0.11095436079645714, pvalue=0.0005848267448603055), men: SpearmanrResult(correlation=0.11504369222990082, pvalue=4.28960553784828e-09), sim353: SpearmanrResult(correlation=0.1318387925484482, pvalue=0.01867078025276011), nearest to monster: ['monster', 'blade', 'tail', 'boat', 'nickname', 'owner', 'leg', 'plain', 'pole', 'ghost']
epoch: 0, iter: 64100, loss: 30.403200149536133
epoch: 0, iter: 64200, loss: 31.01869010925293
epoch: 0, iter: 64300, loss: 30.85900115966797
epoch: 0, iter: 64400, loss: 31.06339454650879
epoch: 0, iter: 64500, loss: 31.443498611450195
epoch: 0, iter: 64600, loss: 30.922685623168945
epoch: 0, iter: 64700, loss: 30.92823028564453
epoch: 0, iter: 64800, loss: 30.95685577392578
epoch: 0, iter: 64900, loss: 31.249370574951172
epoch: 0, iter: 65000, loss: 31.283973693847656
epoch: 0, iter: 65100, loss: 31.421056747436523
epoch: 0, iter: 65200, loss: 31.271799087524414
epoch: 0, iter: 65300, loss: 31.055686950683594
epoch: 0, iter: 65400, loss: 31.06484603881836
epoch: 0, iter: 65500, loss: 31.523380279541016
epoch: 0, iter: 65600, loss: 30.86985969543457
epoch: 0, iter: 65700, loss: 31.431381225585938
epoch: 0, iter: 65800, loss: 30.828258514404297
epoch: 0, iter: 65900, loss: 30.777324676513672
epoch: 0, iter: 66000, loss: 30.793434143066406
epoch: 0, iteration: 66000, simlex-999: SpearmanrResult(correlation=0.11336484593643655, pvalue=0.000441848134889879), men: SpearmanrResult(correlation=0.11607771416192764, pvalue=3.108529516895944e-09), sim353: SpearmanrResult(correlation=0.13567689693977825, pvalue=0.015471766603603733), nearest to monster: ['monster', 'blade', 'tail', 'boat', 'plain', 'mouse', 'leg', 'dubbed', 'angel', 'signature']
epoch: 0, iter: 66100, loss: 30.86065101623535
epoch: 0, iter: 66200, loss: 30.876815795898438
epoch: 0, iter: 66300, loss: 31.069660186767578
epoch: 0, iter: 66400, loss: 30.777523040771484
epoch: 0, iter: 66500, loss: 31.19533920288086
epoch: 0, iter: 66600, loss: 30.554855346679688
epoch: 0, iter: 66700, loss: 30.99230194091797
epoch: 0, iter: 66800, loss: 31.07242202758789
epoch: 0, iter: 66900, loss: 30.73615264892578
epoch: 0, iter: 67000, loss: 31.139455795288086
epoch: 0, iter: 67100, loss: 30.713550567626953
epoch: 0, iter: 67200, loss: 31.377769470214844
epoch: 0, iter: 67300, loss: 31.406234741210938
epoch: 0, iter: 67400, loss: 30.695165634155273
epoch: 0, iter: 67500, loss: 31.24422836303711
epoch: 0, iter: 67600, loss: 30.47709083557129
epoch: 0, iter: 67700, loss: 30.54576301574707
epoch: 0, iter: 67800, loss: 31.31440544128418
epoch: 0, iter: 67900, loss: 30.91347312927246
epoch: 0, iter: 68000, loss: 31.376529693603516
epoch: 0, iteration: 68000, simlex-999: SpearmanrResult(correlation=0.11452703856698428, pvalue=0.00038523421818805733), men: SpearmanrResult(correlation=0.11616177986176482, pvalue=3.027823236694649e-09), sim353: SpearmanrResult(correlation=0.13998669447166648, pvalue=0.012461259148780957), nearest to monster: ['monster', 'boat', 'blade', 'signature', 'tail', 'dubbed', 'mouse', 'cave', 'angel', 'replacing']
epoch: 0, iter: 68100, loss: 30.246761322021484
epoch: 0, iter: 68200, loss: 31.031679153442383
epoch: 0, iter: 68300, loss: 31.415462493896484
epoch: 0, iter: 68400, loss: 30.809803009033203
epoch: 0, iter: 68500, loss: 31.034048080444336
epoch: 0, iter: 68600, loss: 31.134660720825195
epoch: 0, iter: 68700, loss: 31.344093322753906
epoch: 0, iter: 68800, loss: 31.488487243652344
epoch: 0, iter: 68900, loss: 31.44832992553711
epoch: 0, iter: 69000, loss: 30.69766616821289
epoch: 0, iter: 69100, loss: 31.20623016357422
epoch: 0, iter: 69200, loss: 31.305984497070312
epoch: 0, iter: 69300, loss: 30.8837947845459
epoch: 0, iter: 69400, loss: 30.787147521972656
epoch: 0, iter: 69500, loss: 30.73443603515625
epoch: 0, iter: 69600, loss: 30.5230712890625
epoch: 0, iter: 69700, loss: 30.885122299194336
epoch: 0, iter: 69800, loss: 30.608633041381836
epoch: 0, iter: 69900, loss: 31.044784545898438
epoch: 0, iter: 70000, loss: 30.79353141784668
epoch: 0, iteration: 70000, simlex-999: SpearmanrResult(correlation=0.11447412364241306, pvalue=0.00038765739483096796), men: SpearmanrResult(correlation=0.11756886224004129, pvalue=1.944056651206125e-09), sim353: SpearmanrResult(correlation=0.1422861985318877, pvalue=0.011076632617477473), nearest to monster: ['monster', 'boat', 'mouse', 'blade', 'angel', 'replacing', 'leg', 'signature', 'legendary', 'tail']
epoch: 0, iter: 70100, loss: 31.068965911865234
epoch: 0, iter: 70200, loss: 31.09067153930664
epoch: 0, iter: 70300, loss: 30.815410614013672
epoch: 0, iter: 70400, loss: 31.200820922851562
epoch: 0, iter: 70500, loss: 30.970481872558594
epoch: 0, iter: 70600, loss: 30.677066802978516
epoch: 0, iter: 70700, loss: 31.553955078125
epoch: 0, iter: 70800, loss: 30.71514892578125
epoch: 0, iter: 70900, loss: 30.628828048706055
epoch: 0, iter: 71000, loss: 30.579029083251953
epoch: 0, iter: 71100, loss: 30.71631622314453
epoch: 0, iter: 71200, loss: 31.383621215820312
epoch: 0, iter: 71300, loss: 30.815113067626953
epoch: 0, iter: 71400, loss: 31.219520568847656
epoch: 0, iter: 71500, loss: 30.879241943359375
epoch: 0, iter: 71600, loss: 30.864967346191406
epoch: 0, iter: 71700, loss: 31.130123138427734
epoch: 0, iter: 71800, loss: 31.275012969970703
epoch: 0, iter: 71900, loss: 30.728538513183594
epoch: 0, iter: 72000, loss: 30.295608520507812
epoch: 0, iteration: 72000, simlex-999: SpearmanrResult(correlation=0.11509252437598057, pvalue=0.00036020627011201973), men: SpearmanrResult(correlation=0.11721625827550092, pvalue=2.173406746828479e-09), sim353: SpearmanrResult(correlation=0.15003741548009542, pvalue=0.007357771752434236), nearest to monster: ['monster', 'boat', 'leg', 'angel', 'replacing', 'pole', 'legendary', 'tail', 'mouse', 'signature']
epoch: 0, iter: 72100, loss: 30.66713523864746
epoch: 0, iter: 72200, loss: 30.61351776123047
epoch: 0, iter: 72300, loss: 31.320636749267578
epoch: 0, iter: 72400, loss: 31.034809112548828
epoch: 0, iter: 72500, loss: 31.062036514282227
epoch: 0, iter: 72600, loss: 30.442829132080078
epoch: 0, iter: 72700, loss: 30.91510581970215
epoch: 0, iter: 72800, loss: 30.70620346069336
epoch: 0, iter: 72900, loss: 30.421703338623047
epoch: 0, iter: 73000, loss: 30.53826141357422
epoch: 0, iter: 73100, loss: 30.770679473876953
epoch: 0, iter: 73200, loss: 31.04900360107422
epoch: 0, iter: 73300, loss: 30.795854568481445
epoch: 0, iter: 73400, loss: 31.299104690551758
epoch: 0, iter: 73500, loss: 30.484947204589844
epoch: 0, iter: 73600, loss: 30.79161834716797
epoch: 0, iter: 73700, loss: 30.636621475219727
epoch: 0, iter: 73800, loss: 31.00129508972168
epoch: 0, iter: 73900, loss: 30.91973114013672
epoch: 0, iter: 74000, loss: 31.55290985107422
epoch: 0, iteration: 74000, simlex-999: SpearmanrResult(correlation=0.11672803148915531, pvalue=0.0002961005658581428), men: SpearmanrResult(correlation=0.11817601695076835, pvalue=1.6031687449902205e-09), sim353: SpearmanrResult(correlation=0.15298232562148392, pvalue=0.006267834790300931), nearest to monster: ['monster', 'angel', 'legendary', 'leg', 'boat', 'replacing', 'tail', 'dubbed', 'cave', 'pole']
epoch: 0, iter: 74100, loss: 31.169639587402344
epoch: 0, iter: 74200, loss: 30.829368591308594
epoch: 0, iter: 74300, loss: 30.788063049316406
epoch: 0, iter: 74400, loss: 30.632108688354492
epoch: 0, iter: 74500, loss: 30.72389030456543
epoch: 0, iter: 74600, loss: 30.648719787597656
epoch: 0, iter: 74700, loss: 31.583736419677734
epoch: 0, iter: 74800, loss: 30.765384674072266
epoch: 0, iter: 74900, loss: 30.931472778320312
epoch: 0, iter: 75000, loss: 30.993127822875977
epoch: 0, iter: 75100, loss: 30.643539428710938
epoch: 0, iter: 75200, loss: 30.458595275878906
epoch: 0, iter: 75300, loss: 30.298744201660156
epoch: 0, iter: 75400, loss: 30.824840545654297
epoch: 0, iter: 75500, loss: 31.22673797607422
epoch: 0, iter: 75600, loss: 30.644241333007812
epoch: 0, iter: 75700, loss: 30.66327667236328
epoch: 0, iter: 75800, loss: 31.115676879882812
epoch: 0, iter: 75900, loss: 30.466846466064453
epoch: 0, iter: 76000, loss: 30.9396915435791
epoch: 0, iteration: 76000, simlex-999: SpearmanrResult(correlation=0.11596381543200739, pvalue=0.00032459762253928627), men: SpearmanrResult(correlation=0.11879493045996324, pvalue=1.3158101883470934e-09), sim353: SpearmanrResult(correlation=0.15550872005311292, pvalue=0.005450575993741813), nearest to monster: ['monster', 'angel', 'blade', 'replacing', 'boat', 'legendary', 'dubbed', 'leg', 'pole', 'tail']
epoch: 0, iter: 76100, loss: 30.884302139282227
epoch: 0, iter: 76200, loss: 30.992034912109375
epoch: 0, iter: 76300, loss: 30.93535041809082
epoch: 0, iter: 76400, loss: 31.227296829223633
epoch: 0, iter: 76500, loss: 30.600688934326172
epoch: 0, iter: 76600, loss: 30.734973907470703
epoch: 0, iter: 76700, loss: 31.285720825195312
epoch: 0, iter: 76800, loss: 30.783761978149414
epoch: 0, iter: 76900, loss: 31.069557189941406
epoch: 0, iter: 77000, loss: 31.12335205078125
epoch: 0, iter: 77100, loss: 30.547470092773438
epoch: 0, iter: 77200, loss: 30.63747215270996
epoch: 0, iter: 77300, loss: 30.923892974853516
epoch: 0, iter: 77400, loss: 30.970041275024414
epoch: 0, iter: 77500, loss: 31.035385131835938
epoch: 0, iter: 77600, loss: 30.704097747802734
epoch: 0, iter: 77700, loss: 30.910247802734375
epoch: 0, iter: 77800, loss: 30.7044734954834
epoch: 0, iter: 77900, loss: 30.660982131958008
epoch: 0, iter: 78000, loss: 30.560914993286133
epoch: 0, iteration: 78000, simlex-999: SpearmanrResult(correlation=0.11666061112723143, pvalue=0.00029851725007368435), men: SpearmanrResult(correlation=0.12038941573817902, pvalue=7.873392979563775e-10), sim353: SpearmanrResult(correlation=0.15946717336677685, pvalue=0.004361303455838544), nearest to monster: ['monster', 'angel', 'legendary', 'blade', 'boat', 'leg', 'replacing', 'signature', 'tail', 'epic']
epoch: 0, iter: 78100, loss: 31.013874053955078
epoch: 0, iter: 78200, loss: 30.866724014282227
epoch: 0, iter: 78300, loss: 30.68968391418457
epoch: 0, iter: 78400, loss: 31.183582305908203
epoch: 0, iter: 78500, loss: 31.27776336669922
epoch: 0, iter: 78600, loss: 30.777645111083984
epoch: 0, iter: 78700, loss: 31.012584686279297
epoch: 0, iter: 78800, loss: 31.096542358398438
epoch: 0, iter: 78900, loss: 31.362171173095703
epoch: 0, iter: 79000, loss: 30.738359451293945
epoch: 0, iter: 79100, loss: 31.230934143066406
epoch: 0, iter: 79200, loss: 31.539630889892578
epoch: 0, iter: 79300, loss: 30.64007568359375
epoch: 0, iter: 79400, loss: 30.633243560791016
epoch: 0, iter: 79500, loss: 30.83672332763672
epoch: 0, iter: 79600, loss: 30.83807945251465
epoch: 0, iter: 79700, loss: 30.863502502441406
epoch: 0, iter: 79800, loss: 31.011892318725586
epoch: 0, iter: 79900, loss: 30.918609619140625
epoch: 0, iter: 80000, loss: 30.889875411987305
epoch: 0, iteration: 80000, simlex-999: SpearmanrResult(correlation=0.11674166145672565, pvalue=0.0002956142280039947), men: SpearmanrResult(correlation=0.12076227513300175, pvalue=6.975679989657089e-10), sim353: SpearmanrResult(correlation=0.16263542871751985, pvalue=0.00363550206573609), nearest to monster: ['monster', 'blade', 'boat', 'leg', 'legendary', 'bird', 'angel', 'tail', 'signature', 'replacing']
epoch: 0, iter: 80100, loss: 30.875946044921875
epoch: 0, iter: 80200, loss: 31.290252685546875
epoch: 0, iter: 80300, loss: 30.575260162353516
epoch: 0, iter: 80400, loss: 31.11186981201172
epoch: 0, iter: 80500, loss: 30.141450881958008
epoch: 0, iter: 80600, loss: 30.628923416137695
epoch: 0, iter: 80700, loss: 29.730871200561523
epoch: 0, iter: 80800, loss: 30.972171783447266
epoch: 0, iter: 80900, loss: 30.97983169555664
epoch: 0, iter: 81000, loss: 31.120412826538086
epoch: 0, iter: 81100, loss: 31.14563751220703
epoch: 0, iter: 81200, loss: 30.718021392822266
epoch: 0, iter: 81300, loss: 31.257009506225586
epoch: 0, iter: 81400, loss: 30.679397583007812
epoch: 0, iter: 81500, loss: 30.84437370300293
epoch: 0, iter: 81600, loss: 31.431678771972656
epoch: 0, iter: 81700, loss: 30.983497619628906
epoch: 0, iter: 81800, loss: 30.411563873291016
epoch: 0, iter: 81900, loss: 30.98554801940918
epoch: 0, iter: 82000, loss: 30.836700439453125
epoch: 0, iteration: 82000, simlex-999: SpearmanrResult(correlation=0.11957387123233242, pvalue=0.0002092803818694985), men: SpearmanrResult(correlation=0.12247627027840459, pvalue=3.979672482221647e-10), sim353: SpearmanrResult(correlation=0.16490918370598598, pvalue=0.0031840146668676477), nearest to monster: ['monster', 'blade', 'bird', 'boat', 'leg', 'legendary', 'angel', 'tail', 'signature', 'brand']
epoch: 0, iter: 82100, loss: 31.439632415771484
epoch: 0, iter: 82200, loss: 31.005916595458984
epoch: 0, iter: 82300, loss: 30.159162521362305
epoch: 0, iter: 82400, loss: 31.28647232055664
epoch: 0, iter: 82500, loss: 31.14252281188965
epoch: 0, iter: 82600, loss: 30.4515323638916
epoch: 0, iter: 82700, loss: 31.12192153930664
epoch: 0, iter: 82800, loss: 31.36789894104004
epoch: 0, iter: 82900, loss: 30.705039978027344
epoch: 0, iter: 83000, loss: 30.586198806762695
epoch: 0, iter: 83100, loss: 30.898677825927734
epoch: 0, iter: 83200, loss: 30.465381622314453
epoch: 0, iter: 83300, loss: 30.524826049804688
epoch: 0, iter: 83400, loss: 30.76988410949707
epoch: 0, iter: 83500, loss: 31.108976364135742
epoch: 0, iter: 83600, loss: 30.428987503051758
epoch: 0, iter: 83700, loss: 31.005210876464844
epoch: 0, iter: 83800, loss: 30.407583236694336
epoch: 0, iter: 83900, loss: 30.31291961669922
epoch: 0, iter: 84000, loss: 30.384244918823242
epoch: 0, iteration: 84000, simlex-999: SpearmanrResult(correlation=0.11985731732845852, pvalue=0.00020208569630055274), men: SpearmanrResult(correlation=0.12383970685303505, pvalue=2.5324734087303694e-10), sim353: SpearmanrResult(correlation=0.16816942668263687, pvalue=0.002625098004568829), nearest to monster: ['monster', 'blade', 'leg', 'angel', 'boat', 'legendary', 'bird', 'tail', 'signature', 'epic']
epoch: 0, iter: 84100, loss: 30.388629913330078
epoch: 0, iter: 84200, loss: 30.61890983581543
epoch: 0, iter: 84300, loss: 30.860755920410156
epoch: 0, iter: 84400, loss: 30.6082763671875
epoch: 0, iter: 84500, loss: 30.51343536376953
epoch: 0, iter: 84600, loss: 31.175479888916016
epoch: 0, iter: 84700, loss: 30.97124481201172
epoch: 0, iter: 84800, loss: 30.84918975830078
epoch: 0, iter: 84900, loss: 30.95672035217285
epoch: 0, iter: 85000, loss: 31.12570571899414
epoch: 0, iter: 85100, loss: 31.057252883911133
epoch: 0, iter: 85200, loss: 30.39339828491211
epoch: 0, iter: 85300, loss: 30.523571014404297
epoch: 0, iter: 85400, loss: 30.765701293945312
epoch: 0, iter: 85500, loss: 30.65972137451172
epoch: 0, iter: 85600, loss: 30.2365779876709
epoch: 0, iter: 85700, loss: 31.060688018798828
epoch: 0, iter: 85800, loss: 31.084121704101562
epoch: 0, iter: 85900, loss: 30.77812957763672
epoch: 0, iter: 86000, loss: 30.55185890197754
epoch: 0, iteration: 86000, simlex-999: SpearmanrResult(correlation=0.12072190676944367, pvalue=0.00018154682975915078), men: SpearmanrResult(correlation=0.1252523395746619, pvalue=1.577244824410371e-10), sim353: SpearmanrResult(correlation=0.1690460146471711, pvalue=0.002490881483585671), nearest to monster: ['monster', 'blade', 'leg', 'angel', 'boat', 'tail', 'bird', 'mirror', 'legendary', 'signature']
epoch: 0, iter: 86100, loss: 30.656890869140625
epoch: 0, iter: 86200, loss: 30.80274200439453
epoch: 0, iter: 86300, loss: 30.992799758911133
epoch: 0, iter: 86400, loss: 30.460365295410156
epoch: 0, iter: 86500, loss: 30.55353546142578
epoch: 0, iter: 86600, loss: 31.388164520263672
epoch: 0, iter: 86700, loss: 30.856948852539062
epoch: 0, iter: 86800, loss: 30.76443099975586
epoch: 0, iter: 86900, loss: 30.570655822753906
epoch: 0, iter: 87000, loss: 30.948423385620117
epoch: 0, iter: 87100, loss: 30.856409072875977
epoch: 0, iter: 87200, loss: 30.930587768554688
epoch: 0, iter: 87300, loss: 30.785308837890625
epoch: 0, iter: 87400, loss: 30.77594757080078
epoch: 0, iter: 87500, loss: 30.602954864501953
epoch: 0, iter: 87600, loss: 31.219999313354492
epoch: 0, iter: 87700, loss: 30.640804290771484
epoch: 0, iter: 87800, loss: 31.12940788269043
epoch: 0, iter: 87900, loss: 30.826904296875
epoch: 0, iter: 88000, loss: 30.990097045898438
epoch: 0, iteration: 88000, simlex-999: SpearmanrResult(correlation=0.12147024702674274, pvalue=0.0001653680990202469), men: SpearmanrResult(correlation=0.12604951290417668, pvalue=1.2045725972710165e-10), sim353: SpearmanrResult(correlation=0.1693454875469321, pvalue=0.002446480354677961), nearest to monster: ['monster', 'blade', 'leg', 'angel', 'bird', 'boat', 'signature', 'legendary', 'mirror', 'owner']
epoch: 0, iter: 88100, loss: 30.87779426574707
epoch: 0, iter: 88200, loss: 30.53211212158203
epoch: 0, iter: 88300, loss: 30.86421012878418
epoch: 0, iter: 88400, loss: 30.66036605834961
epoch: 0, iter: 88500, loss: 30.340608596801758
epoch: 0, iter: 88600, loss: 30.53420639038086
epoch: 0, iter: 88700, loss: 31.032270431518555
epoch: 0, iter: 88800, loss: 30.652175903320312
epoch: 0, iter: 88900, loss: 31.2420654296875
epoch: 0, iter: 89000, loss: 31.169876098632812
epoch: 0, iter: 89100, loss: 30.760807037353516
epoch: 0, iter: 89200, loss: 31.122560501098633
epoch: 0, iter: 89300, loss: 30.895538330078125
epoch: 0, iter: 89400, loss: 30.56373405456543
epoch: 0, iter: 89500, loss: 30.996185302734375
epoch: 0, iter: 89600, loss: 30.380939483642578
epoch: 0, iter: 89700, loss: 31.11984634399414
epoch: 0, iter: 89800, loss: 30.738248825073242
epoch: 0, iter: 89900, loss: 30.822444915771484
epoch: 0, iter: 90000, loss: 31.190614700317383
epoch: 0, iteration: 90000, simlex-999: SpearmanrResult(correlation=0.12146351761533054, pvalue=0.00016550735138900002), men: SpearmanrResult(correlation=0.12797080964385293, pvalue=6.246963375652522e-11), sim353: SpearmanrResult(correlation=0.1730852603381537, pvalue=0.0019495550082259915), nearest to monster: ['monster', 'blade', 'leg', 'angel', 'signature', 'bird', 'boat', 'tail', 'legendary', 'mirror']
epoch: 0, iter: 90100, loss: 31.01602554321289
epoch: 0, iter: 90200, loss: 30.99297523498535
epoch: 0, iter: 90300, loss: 31.247032165527344
在 MEN 和 Simplex-999 数据集上做评估
embedding_weights = model.input_embeddings()
print("simlex-999", evaluate("simlex-999.txt", embedding_weights))
print("men", evaluate("men.txt", embedding_weights))
print("wordsim353", evaluate("wordsim353.csv", embedding_weights))simlex-999 SpearmanrResult(correlation=0.17251697429101504, pvalue=7.863946056740345e-08)
men SpearmanrResult(correlation=0.1778096817088841, pvalue=7.565661657312768e-20)
wordsim353 SpearmanrResult(correlation=0.27153702278146635, pvalue=8.842165885381714e-07)
寻找nearest neighbors
for word in ["good", "fresh", "monster", "green", "like", "america", "chicago", "work", "computer", "language"]:
print(word, find_nearest(word))good ['good', 'bad', 'perfect', 'hard', 'questions', 'alone', 'money', 'false', 'truth', 'experience']
fresh ['fresh', 'grain', 'waste', 'cooling', 'lighter', 'dense', 'mild', 'sized', 'warm', 'steel']
monster ['monster', 'giant', 'robot', 'hammer', 'clown', 'bull', 'demon', 'triangle', 'storyline', 'slogan']
green ['green', 'blue', 'yellow', 'white', 'cross', 'orange', 'black', 'red', 'mountain', 'gold']
like ['like', 'unlike', 'etc', 'whereas', 'animals', 'soft', 'amongst', 'similarly', 'bear', 'drink']
america ['america', 'africa', 'korea', 'india', 'australia', 'turkey', 'pakistan', 'mexico', 'argentina', 'carolina']
chicago ['chicago', 'boston', 'illinois', 'texas', 'london', 'indiana', 'massachusetts', 'florida', 'berkeley', 'michigan']
work ['work', 'writing', 'job', 'marx', 'solo', 'label', 'recording', 'nietzsche', 'appearance', 'stage']
computer ['computer', 'digital', 'electronic', 'audio', 'video', 'graphics', 'hardware', 'software', 'computers', 'program']
language ['language', 'languages', 'alphabet', 'arabic', 'grammar', 'pronunciation', 'dialect', 'programming', 'chinese', 'spelling']
单词之间的关系
vec_women - vec_man = vec_??? - vec_king
man_idx = word_to_idx["man"]
king_idx = word_to_idx["king"]
woman_idx = word_to_idx["woman"]
embedding = embedding_weights[woman_idx] - embedding_weights[man_idx] + embedding_weights[king_idx]
cos_dis = np.array([scipy.spatial.distance.cosine(e, embedding) for e in embedding_weights])
for i in cos_dis.argsort()[:20]:
print(idx_to_word[i])
# 评估部分都没有仔细看。king
henry
charles
pope
queen
iii
prince
elizabeth
alexander
constantine
edward
son
iv
louis
emperor
mary
james
joseph
frederick
francis