[Time Series Forecasting] Introduction to Nbeats series models

Today I will introduce the Nbeats series of models for time series forecasting published by ElementAI, a unicorn company created by Turing Award winner Bengio. Compared with the three mainstream time prediction model frameworks of RNN, transformer, and CNN, Nbeats has created a new backbone, which uses only fully connected networks to achieve time series prediction.

N-BEATS: Neural basis expansion analysis for interpretable time series forecasting

Document address: https://arxiv.org/pdf/1905.10437.pdf
Code address: https://github.com/jdb78/pytorch-forecasting/blob/master/examples/nbeats.py
The main idea

The N-BEATS model solves the task of unary time series prediction, that is, given a historical sequence of length T, predict its values within H windows in the future. The author of the article claims that the N-BEATS model framework follows three basic principles. First, the model framework is simple and universal while having expressive power (depth). Second, the model architecture does not depend on specific time-series feature engineering or input scaling, thereby exploring the potential of deep learning model structures for time-series forecasting. Finally, the model architecture is interpretive. Without further ado, let’s introduce the N-BEATS model structure in detail below.

(From right to left are the basic modules of N-BEATS)

The N-BEATS model is composed of multiple stack modules, and each stack is composed of multiple blocks in series. The input of the first block is the original input sequence, and the output is two parts, one is the predicted value for the future window H, The second is the reconstructed value of the block input. The inputs of the remaining blocks are all the residuals of the previous block input minus the reconstructed value of the previous block output. On the one hand, the next block is used as a supplement to the previous block to continuously fit the residual information that the previous block did not fit. On the other hand, this process can be regarded as the decomposition of time series information. In different stacks, The block fits the information of a certain part of the time series, and the final output of Nbeats is the sum of the output of each stack.

The output consists of two parts, one is the predicted value for the future window H, and the other is the reconstructed value for the block input. For each block, it contains two branches for and, and each branch can be regarded as a two-stage mapping process. The first stage of mapping maps the input sequence to expansion coefficient (or), including 4 FC layers ( It consists of a fully connected layer and a RELU activation function) and a simple linear layer, as follows:

The expansion coefficient can be understood as a low-dimensional vector that retains the predicted target or closely related information after the block removes the redundant information in the input. The mapping in the second stage is to sum the expansion coefficient obtained in the previous stage through the mapping function to obtain the sum. The example is as follows

The mapping function can be set to the general form above, with learnable coefficients, or it can be set to a specific function form to force the blocks in a stack to learn specific attributes of the time series to enhance the interpretability of the N-BEATS model sex. For example, it can be set as the following form, so that the block can learn the trend information and seasonal information of the time series respectively.

NBEATSx ：Neural basis expansion analysis with exogenous variables: Forecasting electricity prices with NBEATSx

Document address: https://arxiv.org/pdf/2104.05522.pdf
Code address: https://github.com/cchallu/nbeatsx
The main idea

NBEATSx expands N-BEATS and improves in two aspects. 1) In addition to time series, it allows the input of external covariates, so as to extract useful information and time-dependent information in the data. 2) Interpretable time series signal decomposition, flexible fitting of various seasonality in time series through nonlinear transformation, and interaction of factors such as holidays and external covariates.

Like N-BEATS, NBEATSx is also composed of stacks and blocks. The main difference is in the block part. In addition to the historical sequence, the input of the block also contains external covariates, which are transformed into hidden states through the fully connected layer, and then transformed into the expansion coefficient sum through two linear layers. At this time, the expansion coefficient sum retains the information in the external covariates and historical series.

After obtaining the expansion coefficient, the next step is to perform mapping to obtain the predicted sum of past values and future values. In order to improve the interpretability of the model, different stacks can use harmonic functions, polynomial trend items and external covariates to respectively fit the information related to the period, trend and external information in the sequence, as follows:

Of course, NBEATSx can also use a general mapping function, and further proposes a new mapping function, that is, using a TCN (time series convolution module) to encode the external features, and use the encoding as the mapping coefficient to generate the prediction result, so as to capture Long-run dependencies among covariates.

FC-GAGA: Fully Connected Gated Graph Architecture for Spatio-Temporal Traffic Forecasting

Document address: https://arxiv.org/pdf/2007.15531.pdf
Code address: https://github.com/boreshkinai/fc-gaga
The main idea

N-BEATS is mainly oriented to the problem of unary time series prediction. FC-GAGA combines the N-BEATS model with the graph gate mechanism to model the mutual influence relationship between multivariate time series and solve the problem of multivariate time series prediction. It is similar to NBEATSx , FC-GAGA also allows the input of external covariates.

Multivariate time series can be described by a graph. Each sequence can be regarded as a node in the graph, which is generated by an unknown random process. The edges between nodes can describe the influence relationship between sequences. The goal of FC-GAGA is to predict each The value of the node in the future H window. The basic module of FC-GAGA is as shown in the figure above. Three new steps have been added to the stack module of Nbeats, namely Graph edge weights, Time gate block, and Graph gate block.

Graph edge weights: Since the relationship between variables is unknown in advance, it is first necessary to obtain the connection graph between variables. Encode the eigenvectors of each node to obtain the d-dimensional node embedding. By calculating the inner product between the two node embeddings, the adjacency matrix W representing the relationship between the two nodes can be obtained:

ε is a scaling factor.

Time gate block : This module uses the multiplicative gate model shown in the purple module in the picture. The module input splices time-related covariates (such as day of the week, time) and historical sequences to fit different time-related patterns of each node. This module aims to eliminate the unique multiplicative seasonality in each node and multiply it with the predicted value of each node on output to recover the pattern of each node. Because the time corresponding to the input and output sequences is usually different, through this method, the impact of time is fully considered, time-related features can be better modeled, possible seasonal differences between nodes are considered, and enhanced Model flexibility.

Graph gate block : This module relates information between different nodes based on the size of the connection relationship between nodes. For the input historical data matrix, first calculate the maximum value of each node, and then plan the following operation:

Obtain a matrix, each row of the matrix corresponds to a node. The above operation realizes the interaction between node i and node j at each time value of history w. Through the relu function and the gating mechanism, the influence between pairs of unrelated nodes is eliminated, and the size of the node affects the gating mechanism. However, the author did not explain too much why the maximum value of the node needs to be subtracted and removed.

Finally, the matrix is input into the block of Nbeats as the updated sequence value to predict the historical value and future value. The predicted future value is multiplied by the output of the time gate block to obtain the final output value of this layer. The final predicted value is obtained by accumulating multi-layer results.

Summarize

The Nbeats series models use the fully connected layer as the backbone to realize the prediction of unary time series and the space-time prediction of multivariate time series. The core idea is to use the fully connected layer to decompose the time series, and to model the periodic items of the time series through different functional forms. Trend terms, etc., improve the interpretability of the model, and the model structure is relatively simple.