Simple RNN modle
循环神经网络,主要用于挖掘数据中的时序信息以及语义信息的深度表达能力,在语音识别,语言模型,机器翻译以及时序分析方面也被广泛应用.举个例子,比如文本序列的预测,预测句子的下一个单词是什么,一般需要当前的单词以及前面的单词,因为句子的各之间不独立的,比如当前单词是is,前一个词汇是sky,那么下一个词汇很大的概率是blue,RNN就是通过对大量的序列数据的学习,网络会记忆之前的信息,并依据之前的信息来推测后来的输出信息.所以在RNN中隐藏层节点之间是有连接的,隐藏层当前的状态是由但前和输入和上一个时间点的隐藏层(状态)输出共同决定的.
一个简单的文本生成模型
训练文本
import necessay modules
from __future__ import print_functionfrom keras.layers import Dense, Activation
from keras.layers.recurrent import SimpleRNN
from keras.models import Sequential
from keras.utils.vis_utils import plot_model
import numpy as np训练文本是一短英文文本,包含换行符和一些非ASCII字符,训练数据是基于字符的并不是一个完整的单词,这样比较简单,避免切词,但是如果要进行word-level的流程是差不多一样的.
对文本进行一些简单的清洗
def process_txt(open_path):
with open(open_path, 'rb') as f:
lines = []
for line in f:
line = line.strip().lower()
line = line.decode("ascii", "ignore")
if 0 == len(line):
continue
lines.append(line)
text = ' '.join(lines)
return texttext = process_txt('test.txt')len(text)1274
building a character-level RNN
对字符在字典中的位置进行编码,存储在字典对象中
chars = set([c for c in text])
chars_count = len(chars)
char2index = dict((c, i) for i, c in enumerate(chars))
index2char = dict((i, c) for i, c in enumerate(chars))定义变量,STEP 和 SEQLEN
*STEP : 输入序列在整个时间序列数据上的span(跨度),滑动的步长
SEQLEN: 输入序列的长度
举个例子:假设整个序列是abcdefghijklmnopqrstuvw
STEP = 1, SEQLEN = 5
>abcde -> f
>bcdef -> g
>cdefg -> h
>defgh -> i
>efghi -> j
……….
create input and label texts
SEQLEN = 10
STEP = 1
input_chars = []
label_chars = []
for i in range(0, len(text) - SEQLEN, STEP):
input_chars.append(text[i:i+SEQLEN])
label_chars.append(text[i+SEQLEN])vectorize input and label texts
通过one-hot编码来向量化input和label,
X = np.zeros((len(input_chars), SEQLEN, chars_count), dtype=np.bool)
Y = np.zeros((len(input_chars), chars_count), dtype=np.bool)
for i,input_char in enumerate(input_chars):
for j,c in enumerate(input_char):
X[i, j, char2index[c]] = 1
Y[i, char2index[label_chars[i]]] = 1X.shape, Y.shape((1264, 10, 33), (1264, 33))
Build Model
NUM_ITERATIONS : 模型训练的轮数
NUM_EPOCHS_PER_ITERATION : 每隔几轮进行一次测试
NUM_PREDS_PER_EPOCH : 使用模型生成新的字符的次数
给训练好的模型一个随机的时间序列输入,得到一个字符的输出
然后去掉输入的第一个字符,加上那个输出的一个字符,当作新的序列输入
如此这般,并打印输出
HIDDEN_SIZE = 128
BATCH_SIZE = 128
NUM_ITERATIONS = 25
NUM_EPOCHS_PER_ITERATION = 1
NUM_PREDS_PER_EPOCH = 100
model = Sequential()
model.add(SimpleRNN(HIDDEN_SIZE, return_sequences=False,
input_shape=(SEQLEN, chars_count),unroll=True))
model.add(Dense(chars_count))
model.add(Activation("softmax"))
model.compile(loss="categorical_crossentropy", optimizer="rmsprop")Training and predict
for iteration in range(NUM_ITERATIONS):
print('Iteration : %d'%iteration)
model.fit(X, Y, batch_size=BATCH_SIZE, epochs=NUM_EPOCHS_PER_ITERATION)
# 训练1epoch,测试一次
test_idx = np.random.randint(len(input_chars))
test_chars = input_chars[test_idx]
print('test seed is : %s'%test_chars)
print(test_chars,end='')
for i in range(NUM_PREDS_PER_EPOCH):
# 测试序列向量化
vec_test = np.zeros((1, SEQLEN, chars_count))
for i, ch in enumerate(test_chars):
vec_test[0, i, char2index[ch]] = 1
pred = model.predict(vec_test, verbose=0)[0]
pred_char = index2char[np.argmax(pred)]
print(pred_char,end='')
# 不断的加入新生成字符组成新的序列
test_chars = test_chars[1:] + pred_char
print('\n')Iteration : 0
Epoch 1/1
1264/1264 [==============================] - 0s 100us/step - loss: 0.5441
test seed is : d text, wh
d text, where we set the labelas t oh puaciof iin a ie mrcterelice toat word in a text givel the the oaxu is
Iteration : 1
Epoch 1/1
1264/1264 [==============================] - 0s 85us/step - loss: 0.5206
test seed is : based lang
based language model olxt pice live sois aoslainl u cpnedeto anfed tonguert of purdii toreabile ions baelling
Iteration : 2
Epoch 1/1
1264/1264 [==============================] - 0s 87us/step - loss: 0.4940
test seed is : ut x t+1 a
ut x t+1 at time t+1. for our first example of using keras for building ranguage mrdels arclampleta toompstio
Iteration : 3
Epoch 1/1
1264/1264 [==============================] - 0s 90us/step - loss: 0.4797
test seed is : s instead
s instead of words. we till fo to s ahi a tens an uampir t iomsaiie fomelertoor mpreti s omobiin aoesile t
Iteration : 4
Epoch 1/1
1264/1264 [==============================] - 0s 87us/step - loss: 0.4601
test seed is : rate some
rate some text in the same at lsing f nper1bimedeto tfewoed tood thtra te tuaite t od aois cherabtens asetuu-e
Iteration : 5
Epoch 1/1
1264/1264 [==============================] - 0s 94us/step - loss: 0.4285
test seed is : lling corr
lling correction, and so on ta winut irdlle e tha oathris t ha preeicte horrdst s wllla a aseccerneutod the
Iteration : 6
Epoch 1/1
1264/1264 [==============================] - 0s 88us/step - loss: 0.4158
test seed is : typically
typically a sequence of predicted words. the trained model to the tate that atinss uf lire oxe iacs fom ratba
Iteration : 7
Epoch 1/1
1264/1264 [==============================] - 0s 89us/step - loss: 0.4083
test seed is : various h
various higang aereretilus t r sees in sodea ged in taedtin t adrline s aovengexee toxe il t uis tr wards ts
Iteration : 8
Epoch 1/1
1264/1264 [==============================] - 0s 89us/step - loss: 0.3825
test seed is : g a word-b
g a word-based language model anrewe ue toxte word ie a text as aamivel ofsares. anmaaee tooctalini n adele e
Iteration : 9
Epoch 1/1
1264/1264 [==============================] - 0s 106us/step - loss: 0.3636
test seed is : ulary and
ulary and trains quicker. the idea is the same ity s.ng toe mufprt tfxo pseboiis to t illate s s aus d rarali
Iteration : 10
Epoch 1/1
1264/1264 [==============================] - 0s 104us/step - loss: 0.3598
test seed is : nguage mod
nguage models. a language models. a language models. a language models. a language models. a language models.
Iteration : 11
Epoch 1/1
1264/1264 [==============================] - 0s 90us/step - loss: 0.3374
test seed is : o on. a si
o on. a side effect of the ability to predict the next charactersbised aocheracters. ss eae afme ttht so er bu
Iteration : 12
Epoch 1/1
1264/1264 [==============================] - 0s 88us/step - loss: 0.3261
test seed is : instead o
instead of words. we will then use the training data used is enilten oo tuex in ther sme thtu arerabaeil thes
Iteration : 13
Epoch 1/1
1264/1264 [==============================] - 0s 90us/step - loss: 0.3152
test seed is : enerate te
enerate text by ans sevurute th waralt areoatte tho sa inelu wo legtand te tlathe l meael toael the s tuaes to
Iteration : 14
Epoch 1/1
1264/1264 [==============================] - 0s 107us/step - loss: 0.3047
test seed is : (nlp) com
(nlp) community for various applications. one such applications. one such applications. one such applications
Iteration : 15
Epoch 1/1
1264/1264 [==============================] - 0s 108us/step - loss: 0.2944
test seed is : nd the out
nd the output probability of a word in a text given the previous words. language models arelimpo tan for ha i
Iteration : 16
Epoch 1/1
1264/1264 [==============================] - 0s 97us/step - loss: 0.2748
test seed is : ter-based
ter-based model here because it has arsmallertvacabulariens b cherdene s tha term os the tome io t ingtren vro
Iteration : 17
Epoch 1/1
1264/1264 [==============================] - 0s 91us/step - loss: 0.2810
test seed is : wonderlan
wonderland to predict the next character based language models arc amporta thor see iis ay tamllnet. as uach
Iteration : 18
Epoch 1/1
1264/1264 [==============================] - 0s 89us/step - loss: 0.2672
test seed is : te some te
te some text in the same atyle.nu gen whres ten taet axy taralilen uane aoe sifoauel mooe a toc tingta te m1ol
Iteration : 19
Epoch 1/1
1264/1264 [==============================] - 0s 87us/step - loss: 0.2591
test seed is : 1. for our
1. for our first example of using keras for building lnng, ge medel eexe text fivram.min aftf woed bose iod u
Iteration : 20
Epoch 1/1
1264/1264 [==============================] - 0s 86us/step - loss: 0.2424
test seed is : o build a
o build a characters isetead of aoids. whe tuainse moaslli aom abe rd nedtut ve thls ahelasue s oprebivin thed
Iteration : 21
Epoch 1/1
1264/1264 [==============================] - 0s 97us/step - loss: 0.2420
test seed is : on, and so
on, and so on. a side effect of the ability to predict the next character bised language model onxt proeabivit
Iteration : 22
Epoch 1/1
1264/1264 [==============================] - 0s 88us/step - loss: 0.2327
test seed is : is typical
is typically a sequence of predicted words. the trainsd model to tenlrate t mentext of utice tedutaet the tate
Iteration : 23
Epoch 1/1
1264/1264 [==============================] - 0s 87us/step - loss: 0.2186
test seed is : wonderland
wonderland to predict the nrxt chrrdcter biled lang, ge mode praceer ioceaedctoresiton .nn sreg are wordliana
Iteration : 24
Epoch 1/1
1264/1264 [==============================] - 0s 84us/step - loss: 0.2183
test seed is : se charact
se character bised langlage model on the text of alice in wonderland to predict the probabilities. in language