Recently, Hong Kong University of Science and Technology, Shanghai AI Lab and other organizations jointly published an article on time series unsupervised pre-training. Compared with the original time series representation learning work such as TS2Vec, the core is to propose the integration of spatial information into pre-training. stage, that is, the relationship between sequences is considered in the pre-training stage. Therefore, the method proposed in this paper is also more suitable as a pre-training model in the field of spatio-temporal prediction. Let me introduce this article in detail for you.
论文标题：Correlated Time Series Self-Supervised Representation Learning via Spatiotemporal Bootstrapping
Download address : https://arxiv.org/abs/2306.06994
1
background
In the past work, there have been many studies on unsupervised pre-training of time series, and the idea of comparative learning is generally used for self-supervised training of time series Encoder. However, there are three drawbacks to historical work.
The first is that most of the past methods learn the representation of the whole sequence, while the time series prediction task pays more attention to the representation of each time step, so there is a certain mismatch between upstream and downstream tasks.
The second is that past work has been pre-trained on a single time series itself, without considering the relationship between each series.
The third point is that in the previous comparative learning pre-training method, the problem of false negative samples often appeared in the process of constructing negative samples (that is, the constructed negative samples should actually be positive samples), which led to a negative impact on the model effect.
In response to the above three problems, this paper proposes a series of solutions, the core of which is to consider both temporal and spatial relationships in the pre-training stage, and change the comparative learning framework to BYOL that does not rely on positive samples.
2
model details
The core structure of the model proposed in this paper is shown in the figure below. On the one hand, its core is the way to integrate spatio-temporal information into contrastive learning, and on the other hand, it is the upgrade of the contrastive learning framework .
For a time series, it is necessary to conduct comparative learning in both the time dimension and the space dimension, and integrate the spatiotemporal information into the pre-training stage at the same time. For the time dimension, a sliding window method is used in this paper to generate two overlapping subsequences for a sequence, one of which is used as the View and the other as the Target of the time dimension. Among them, the View part uses a random mask to cover a part of the sample points of the time step. Contrastive learning in the time dimension, that is, using comparative learning between View and Temporal Target.
In the spatial dimension, according to the topological relationship between nodes, a neighbor sequence of the current sequence is randomly sampled, and a subsequence is produced by using a sliding window. This sequence is used as the spatial dimension Target of View, and the distance between View and Spatial Target is shortened by using contrastive learning .
After obtaining the above two types of samples, the BYOL comparative learning framework is adopted in this paper, which avoids the construction of negative samples, thus solving the problem of false negative samples . BYOL is a comparative learning framework that only relies on positive samples. The specific method in its original paper is to use two online network and target network with the same model structure but different parameters. The parameters of the target network are the sliding parameters of the online network. On average, the output representation of the target network is the prediction target of the online network, and the target part will not update its parameters according to the loss. That is, two sets of parameters are used to realize the prediction of positive sample 1 to positive sample 2.
In this paper, it is also a similar method, using the partially masked View to predict the Temporal Target and the Spatial Target at the same time. The loss of the model includes the corresponding comparison learning loss of the Temporal and Spatial parts. The specific structure of the model adopts the structure of TCN (the main body is hollow convolution), and the model parameters of View and Target are not shared.
3
Experimental effect
The following is the prediction effect of the representation learning method proposed in this paper on the spatiotemporal prediction dataset. It can be seen that compared with TS2Vec (current SOTA time series representation learning method), there is a more obvious effect improvement. This shows that the contrastive learning that introduces spatial information proposed in this paper is more effective for spatiotemporal prediction types of problems.
The figure below shows the prediction case analysis. Compared with TS2VecUI, the method proposed in this paper has a better prediction trend.
Recommended reading:
My 2022 Internet School Recruitment Sharing
Talking about the difference between algorithm post and development post
Internet school recruitment research and development salary summary
The 2022 Internet job hunting status, gold 9 silver 10 will soon become copper 9 iron 10! !
Public number: AI snail car
Stay humble, stay disciplined, keep improving
Send [Snail] to get a copy of "Hands-on AI Project" (written by AI Snail Car)
Send [1222] to get a good leetcode brushing notes
Send [AI Four Classics] Get four classic AI e-books