illustrate
Memory Networks is a neural network model proposed by Weston last year. It introduces memory effects and external storage, which can effectively deal with long-distance sequence problems. Later, neural Turing machines, neural machine translation machines, etc. appeared in academia. Some models that have attracted very wide interest.
Recently, we are also mainly researching this thing, and this blog is written here as a stage record.
Memory Networks
This project contains implementations of memory networks. This includes code in the following subdirectories:
MemN2N-lang-model: This code trains MemN2N model for language modeling, see Section 5 of the paper “End-To-End Memory Networks”. This code is implemented in Torch7 (written in Lua); more documentation is given in the README in that subdirectory.
MemN2N-babi-matlab: The code for the MemN2N bAbI task experiments of Section 4 of the paper:
S. Sukhbaatar, A. Szlam, J. Weston, R. Fergus. End-To-End Memory Networks. arXiv:1503.08895.
This code is implemented in Matlab; more documentation is given in the README in that subdirectory.
Modification of mine
- The processing scheme of adding unregistered words During the dictionary creation process, only 10 unknown lexemes were added to replace the unregistered words introduced in the test process, using only Train's expectations.
- The training function has been run without softmax (this is very important, otherwise it will not be able to run at all)
- In the corpus of Facebook Q20, which is expected to be diverse, the number of dictionaries is too small to reflect the real logical function. Inconclusive. On the basis of the original Chinese domain-specific dialogue in the Q20 format processed by the seniors, a lot of completely random numbers were added to replace some Chinese characters. And the language has also maintained a variety. Facts have proved that the effect is still good, affirming the memory and reasoning ability of the model.
- There are also some tricks that can only be reflected in the code.
code
https://github.com/shincling/MemNN
result
1 | train error: 0.54656 | val error: 0.12695
2 | train error: 0.13087 | val error: 0.09375
3 | train error: 0.087668 | val error: 0.0625
4 | train error: 0.059983 | val error: 0.048828
5 | train error: 0.037542 | val error: 0.027344
6 | train error: 0.027685 | val error: 0.029297
7 | train error: 0.016359 | val error: 0.029297
8 | train error: 0.0098574 | val error: 0.035156
9 | train error: 0.0079698 | val error: 0.027344
10 | train error: 0.0077601 | val error: 0.029297
11 | train error: 0.0033557 | val error: 0.023438
12 | train error: 0.0020973 | val error: 0.015625
13 | train error: 0.0010487 | val error: 0.0097656
14 | train error: 0.00083893 | val error: 0.011719
15 | train error: 0.00083893 | val error: 0.015625
16 | train error: 0.00062919 | val error: 0.013672
17 | train error: 0.00020973 | val error: 0.0039062
18 | train error: 0 | val error: 0.0039062
19 | train error: 0.00041946 | val error: 0.0058594
20 | train error: 0.00020973 | val error: 0.0058594
test error: 0.0075758
1 | train error: 0 | val error: 0.0039062
2 | train error: 0 | val error: 0.0039062
3 | train error: 0 | val error: 0.0039062
4 | train error: 0 | val error: 0.0019531
5 | train error: 0 | val error: 0.0019531
6 | train error: 0 | val error: 0.0019531
7 | train error: 0 | val error: 0.0019531
8 | train error: 0 | val error: 0.0019531
9 | train error: 0 | val error: 0.0039062
10 | train error: 0 | val error: 0.0058594
11 | train error: 0 | val error: 0.0058594
12 | train error: 0 | val error: 0.0058594
13 | train error: 0 | val error: 0.0039062
14 | train error: 0 | val error: 0.0039062
15 | train error: 0 | val error: 0.0039062
16 | train error: 0 | val error: 0.0039062
17 | train error: 0 | val error: 0.0039062
18 | train error: 0.00062919 | val error: 0.0058594
19 | train error: 0 | val error: 0.0039062
20 | train error: 0 | val error: 0.0039062
21 | train error: 0 | val error: 0.0039062
22 | train error: 0 | val error: 0.0039062
23 | train error: 0 | val error: 0.0039062
24 | train error: 0 | val error: 0.0039062
25 | train error: 0 | val error: 0.0039062
26 | train error: 0 | val error: 0.0039062
27 | train error: 0 | val error: 0.0039062
28 | train error: 0 | val error: 0.0058594
29 | train error: 0 | val error: 0.0058594
30 | train error: 0 | val error: 0.0058594
test error: 0.0066288
1 | train error: 0 | val error: 0.0058594
2 | train error: 0 | val error: 0.0058594
3 | train error: 0 | val error: 0.0058594
4 | train error: 0 | val error: 0.0058594
5 | train error: 0 | val error: 0.0058594
6 | train error: 0 | val error: 0.0058594
7 | train error: 0 | val error: 0.0058594
8 | train error: 0 | val error: 0.0058594
9 | train error: 0 | val error: 0.0058594
10 | train error: 0 | val error: 0.0058594
11 | train error: 0 | val error: 0.0058594
12 | train error: 0 | val error: 0.0058594
13 | train error: 0 | val error: 0.0058594
14 | train error: 0 | val error: 0.0058594
15 | train error: 0 | val error: 0.0058594
16 | train error: 0 | val error: 0.0058594
17 | train error: 0 | val error: 0.0058594
18 | train error: 0 | val error: 0.0058594
19 | train error: 0 | val error: 0.0058594
20 | train error: 0 | val error: 0.0058594
21 | train error: 0 | val error: 0.0058594
22 | train error: 0 | val error: 0.0058594
23 | train error: 0 | val error: 0.0058594
24 | train error: 0 | val error: 0.0058594
25 | train error: 0 | val error: 0.0058594
26 | train error: 0 | val error: 0.0058594
27 | train error: 0 | val error: 0.0058594
28 | train error: 0 | val error: 0.0058594
29 | train error: 0 | val error: 0.0058594
30 | train error: 0 | val error: 0.0058594
test error: 0.0068182
test error: 0.0068182