Keras [Deep Learning With Python] better model to explore Keras achieve LSTM

1.LSTM network

RNN can be understood as an upgrade.
Long Short Term Memory networks (hereinafter referred to as LSTMs), a special RNN network, which is designed to solve the long dependency problems. The network introduced by Hochreiter & Schmidhuber (1997), and many of them have been improved and popularity. Their work has been used to solve a variety of problems, until now also been widely used.

Module chain forms repeating cycle of neural networks have all neural network. RNN in the standard, the repetition module will have a very simple construction, such as a single layer tanh. RNN network standard as shown in FIG

LSTMs also having such a chain structure, but it is different from the repeating unit of the network element RNN standard only a network layer, which has four internal network layer. LSTMs structure as shown below.

In explaining the detailed structure of the first define what we mean by LSTMs drawing each symbol, the symbol comprising the following several

figures like CNN yellow activation function in the operation, the operation point pink circles represent single arrows indicate data flow arrow represents the vector merged the combined (the concat) operation, represented by arrows bifurcated vector copy operation.

2. Before also mentioned RNNs achieved good results, many of these achievements are based on LSTMs do, explain LSTMs for most application scenarios sequence.

3. code implementation

# please note, all tutorial code are running under python3.5.
# If you use the version like python2.7, please modify the code accordingly

# 8 - RNN LSTM Regressor example

# to try tensorflow, un-comment following two lines
# import os
# os.environ['KERAS_BACKEND']='tensorflow'
import numpy as np
np.random.seed(1337)  # for reproducibility
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import LSTM, TimeDistributed, Dense
from keras.optimizers import Adam

BATCH_START = 0
TIME_STEPS = 20
BATCH_SIZE = 50
INPUT_SIZE = 1
OUTPUT_SIZE = 1
CELL_SIZE = 20
LR = 0.006


def get_batch():
    global BATCH_START, TIME_STEPS
    # xs shape (50batch, 20steps)
    xs = np.arange(BATCH_START, BATCH_START+TIME_STEPS*BATCH_SIZE).reshape((BATCH_SIZE, TIME_STEPS)) / (10*np.pi)
    seq = np.sin(xs)
    res = np.cos(xs)
    BATCH_START += TIME_STEPS
    # plt.plot(xs[0, :], res[0, :], 'r', xs[0, :], seq[0, :], 'b--')
    # plt.show()
    return [seq[:, :, np.newaxis], res[:, :, np.newaxis], xs]

model = Sequential()
# build a LSTM RNN
model.add(LSTM(
    batch_input_shape=(BATCH_SIZE, TIME_STEPS, INPUT_SIZE),       # Or: input_dim=INPUT_SIZE, input_length=TIME_STEPS,
    output_dim=CELL_SIZE,
    return_sequences=True,      # True: output at all steps. False: output as last step.
    stateful=True,              # True: the final state of batch1 is feed into the initial state of batch2
))
# add output layer
model.add(TimeDistributed(Dense(OUTPUT_SIZE)))
adam = Adam(LR)
model.compile(optimizer=adam,
              loss='mse',)

print('Training ------------')
for step in range(501):
    # data shape = (batch_num, steps, inputs/outputs)
    X_batch, Y_batch, xs = get_batch()
    cost = model.train_on_batch(X_batch, Y_batch)
    pred = model.predict(X_batch, BATCH_SIZE)
    plt.plot(xs[0, :], Y_batch[0].flatten(), 'r', xs[0, :], pred.flatten()[:TIME_STEPS], 'b--')
    plt.ylim((-1.2, 1.2))
    plt.draw()
    plt.pause(0.1)
    if step % 10 == 0:
        print('train cost: ', cost)