【Paper Introduction】- E-LSTM-D: A Deep Learning Framework for Dynamic Network Link Prediction (Dynamic Network Link Prediction)

Paper information

E-LSTM-D: A Deep Learning Framework for Dynamic Network Link Prediction

原文链接：E-LSTM-D: A Deep Learning Framework for Dynamic Network Link Prediction：https://ieeexplore.ieee.org/abstract/document/8809903

Summary

Predicting the potential relations between nodes in networks, known as link prediction, has long been a challenge in network science. However, most studies just focused on link prediction of static network, while real-world networks always evolve over time with the occurrence and vanishing of nodes and links. Dynamic network link prediction (DNLP) thus has been attracting more and more attention since it can better capture the evolution nature of networks, but still most algorithms fail to achieve satisfied prediction accuracy. Motivated by the excellent performance of long short-term memory (LSTM) in processing time series, in this article, we propose a novel encoder-LSTM-decoder (E-LSTM-D) deep learning model to predict dynamic links end to end. It could handle long-term prediction problems, and suits the networks of different scales with fine-tuned structure. To the best of our knowledge, it is the first time that LSTM, together with an encoder-decoder architecture, is applied to link prediction in dynamic networks. This new model is able to automatically learn structural and temporal features in a unified framework, which can predict the links that never appear in the network before. The extensive experiments show that our E-LSTM-D model significantly outperforms newly proposed DNLP methods and obtain the state-of-the-art results.

Predicting potential relationships between nodes in a network, i.e., link prediction , has long been a challenge in network science. However, most studies only focus on link prediction for static networks, while real-world networks always evolve with the appearance and disappearance of nodes and links. Dynamic network link prediction has attracted more and more attention because it can better capture the evolution characteristics of the network, but most algorithms still cannot achieve satisfactory prediction accuracy. Given the excellent performance of long-short-term memory network LSTM in processing time series, this paper proposes a new encoder-LSTM-decoder (E-LSTM-D) deep learning model to predict end-to-end dynamic links. It can handle long-term forecasting problems and is applicable to networks of different sizes with fine-tuned structures. To the best of our knowledge, this is the first application of LSTMs with encoder-decoder architectures for link prediction in dynamic networks. This new model is able to automatically learn structural and temporal features in a unified framework, which can predict links that have never appeared in the network before. Extensive experiments show that our E-LSTM-D model significantly outperforms the newly proposed DNLP method and achieves state-of-the-art results.

paper contribution

1. A general end-to-end deep learning framework, namely E-LSTM-D, is proposed for link prediction in dynamic networks, where the encoder-decoder architecture automatically learns the representation of the network, and the learning is enhanced by stacked LSTM modules. Ability to characterize time.
2. The newly proposed E-LSTM-D model is capable of long-term prediction tasks with only a slight drop in performance; it is suitable for networks of different scales, by fine-tuning the model structure, that is, changing the number of units in different layers; in addition, it can predict the upcoming or disappearing links, and most existing approaches only focus on the former.
3. A new metric, error rate, is defined to measure the performance of DNLP, which is a nice addition to the area under the ROC curve (AUC), making the evaluation more comprehensive.
4. Extensive experiments are performed to compare the E-LSTM-D model with five baseline methods on various metrics. It shows that our model outperforms other models and achieves state-of-the-art results.

problem definition

dynamic network

Given a sequence of graphs {G1,...,GT}, where Gk = (V, Ek) represents the kth snapshot of a dynamic network. Suppose V is the set of all vertices, Ek∈V × V is the temporal link in the fixed time span [tk-1, tk]. Let Ak represent the adjacency matrix of Gk, and the element is ak; if there is to link, then ak;i,j = 1; otherwise, it is 0.

In static networks, the purpose of link prediction is to find the edges that actually exist based on the observed distribution of edges. Similarly, link prediction in dynamic networks fully exploits the information extracted from previous graphs to reveal the evolution patterns of the underlying network, thereby predicting the future state of the network. Since the adjacency matrix can accurately describe the structure of the network, it is ideal to use it as the input and output of the predictive model. Due to the strong relationship between successive snapshots of the dynamic network, we can infer Gt from Gt − 1. However, the information contained in Gt may be too little to make precise inferences. In fact, in the process of network evolution, not only the structure itself, but also the change of structure over time is equally important. Therefore, we tend to use a sequence of length N, i.e., {Gt-N,...,Gt-1}, to predict Gt.

Dynamic Web Link Prediction

Given a graph sequence S = {Gt-N,...,Gt-1} of length N, DNLP aims to learn a function that maps the input sequence S to Gt.

The structure of a dynamic network evolves over time. As shown in the figure below, some links may appear and others may disappear, which can be reflected by the change of the adjacency matrix over time. Its goal is to find the links in the network that are most likely to appear or disappear in the next time span. Mathematically, it can also be understood as the optimization problem of finding a matrix whose elements are either 0 or 1.

E-LSTM-D framework

E-LSTM-L consists of an encoder-decoder architecture and stacked LSTMs. The encoder is placed at the entrance of the model to learn the highly nonlinear network structure, and the decoder transforms the extracted features back to the original space. Such an encoder-decoder architecture is able to handle spatial nonlinearity and sparsity, while a stacked LSTM between encoder and decoder can learn temporal dependencies. Therefore, a well-designed end-to-end model can simultaneously learn structural and temporal features and perform link prediction in a unified manner.

Encoder–Decoder structure

1. Encoder

Autoencoders can efficiently learn representations of data in an unsupervised manner. Inspired by this, we place an encoder at the entrance of the model to capture the highly nonlinear network structure, and a graph reconstructor at the end to convert the latent features into a fixed-shape matrix. Here, however, the whole process is supervised, unlike autoencoders, because we have labeled the data ( At ) to guide the decoder to build a matrix that better fits the target distribution. In particular, an encoder composed of multiple nonlinear perceptrons projects high-dimensional graphics data into a relatively low-dimensional vector space. Therefore, the resulting vectors can characterize the local structure of vertices in the network. This process can be described as:

where si represents the ith graph in the input sequence S. W(k) e and b(k) e represent the weight and bias of the k-th encoder layer, respectively, and Y ( k ) e is the output of the k-th encoder layer.

For an input sequence, each encoder layer processes each item separately, and then concatenates all activations by element-wise addition.

2. Decoder

The decoder with encoder-mirror structure receives latent features and maps them to the reconstruction space under the supervision of At, denoted as:
where H is generated by a stacked LSTM, denoting the features of the target snapshot, instead of all the features used in the encoder Sequence of features from previous snapshots. W(k)d and b(k)d denote the weights and biases of the kth decoder in the decoder, respectively, and Y(k) denotes the output of the kth decoder layer.

Stacked LSTM

Although the encoder-decoder structure can handle high nonlinearities, it cannot capture time-varying properties. LSTM, as a special kind of RNN, can learn long-term dependencies, and is introduced here to solve this problem. An LSTM consists of three gates, the forget gate, the input gate and the output gate.

1. The first step in the forget gate
   is to decide what information will be discarded from the previous cell state. The operation is performed by the forget gate, which is defined as:
   
2. input gate
   Then the input gate decides which new information should be added to the cell state, the operation is defined as:
   
3. The output gate
   uses the advantages of the forget gate and the input gate, and the LSTM unit can not only store long-term memory, but also filter out useless information. The output of the LSTM unit is based on Ct and is controlled by the output gate, which decides what information should be output. The process is described as:
   
   a single LSTM unit is able to learn temporal dependencies, but chained LSTM modules, i.e. stacked LSTMs, are more suitable for processing time series data. A stacked LSTM consists of multiple LSTM units that take a signal as input in time order. A stacked LSTM is placed between the encoder and decoder to learn patterns of network evolution. After receiving the extracted features at time t, the LSTM module converts them into ht, and then feeds ht back to the model in the next training step.

The encoder can reduce the dimensionality of each graph, keeping the computation of the stacked LSTM at a reasonable cost. The stacked LSTM, which is good at processing sequential and sequential data, complements the encoder in turn.