Summary of Time Series Forecasting Methods: From Theory to Practice (with Kaggle Classic Competition Scheme)

© Author | Light

School | University of Chinese Academy of Sciences

Research direction| Machine learning

Know the link: https://zhuanlan.zhihu.com/p/471014006

Editor | paperweekly

Time series is one of my favorite problems to study, here I list the most commonly used methods of time series , including both theory and practice. Most of the theoretical parts are original high praise interpretations of various gods. Here I simply present them here and attach links. The practical part is high-quality open source code, which is convenient for everyone to get started quickly. Finally, here are some links to the classic solutions of the more classic timing competitions in kaggle competitions for your reference and learning.

Time series problems are regarded as regression problems, but the regression methods (linear regression, tree model, deep learning, etc.) are different.

Traditional Time Series Modeling

The arima model is an upgraded version of the arma model; the arma model can only model stationary data, while the arima model needs to differentiate the data first, and then model the difference after the difference is stationary. The problems that these two models can handle are relatively simple, mainly due to the following two points:

The arma/arima model is still a simple linear model in the final analysis, and the complexity of the problem that can be represented is limited;
The full name of arma is the autoregressive moving average model, which can only support the regression of univariate historical data and cannot handle multivariate situations.

Principles:

Financial Time Series Analysis for You: The Basics

Focus on basic financial time series knowledge and arma models

https://zhuanlan.zhihu.com/p/38320827

Introduction to Financial Time Series [End] ARCH, GARCH

Introduce more advanced arch and garch models

https://zhuanlan.zhihu.com/p/21962996

Practical articles:

[Time series analysis] Python implementation of ARMA forecasting GDP

arma model to get started quickly

https://zhuanlan.zhihu.com/p/54799648

machinelearningmastery.com

Rapid modeling of arch and garch models

https://machinelearningmastery.com/develop-arch-and-garch-models-for-time-series-forecasting-in-python/

Summary: If you are dealing with univariate prediction problems, traditional time series models can play a big advantage; but if there are too many problems or variables, then the traditional time series models will be powerless.

machine learning model approach

This type of method is represented by lightgbm and xgboost, which generally converts time series problems into supervised learning and predicts through feature engineering and machine learning methods; this model can solve most complex time series prediction models. Support complex data modeling, support multivariate collaborative regression, and support nonlinear problems.

However, this method requires a more complex part of the artificial feature process, and feature engineering requires a certain amount of professional knowledge or rich imagination. The level of feature engineering ability often determines the upper limit of machine learning, and machine learning methods only approach this upper limit as much as possible. After the features are established, the tree model algorithm lightgbm/xgboost can be directly applied. These two models are very common rapid modeling methods. In addition, they have the following characteristics:

Fast calculation speed and high model accuracy;
Missing values do not need to be processed, which is more convenient;
Support category variable;
Feature intersection is supported.

Principles:

Lifting tree model: In-depth exploration of Lightgbm principle :

lightgbm principle

https://blog.csdn.net/anshuai_aw1/article/details/83659932

The principle of xgboost is not as difficult as you think :

Principle of xgboost

https://www.jianshu.com/p/7467e616f227

Practical articles:

Using Lightgbm in Python :

lightgbm model practice

https://zhuanlan.zhihu.com/p/52583923

The most detailed XGBoost combat in history :

xgboost model practice

https://zhuanlan.zhihu.com/p/31182879

Summary: After a series of feature engineering, using machine learning methods directly can solve most complex timing problems; however, the biggest disadvantage of this method is that feature engineering may be cumbersome.

Deep Learning Model Methods

Such methods are mainly LSTM/GRU, seq2seq, wavenet, 1D-CNN, and transformer. The LSTM/GRU model in deep learning is specially designed to solve time series problems; but the CNN model was originally designed to solve image problems, but after evolution and development, it can also be used to solve time series problems. In general, deep learning models have the following characteristics:

Missing values cannot be included, and missing values must be filled, otherwise an error will be reported;
Support feature crossover, such as second-order crossover, high-order crossover, etc.;
The embedding layer is required to process category variables, which can directly learn the semantic variables of discrete features and characterize their relative relationships;
When the amount of data is small, the model effect is not as good as the tree method; but when the amount of data is huge, the neural network will perform better;
Neural network models support online training.

In fact, a wide variety of deep learning model architectures can be designed based on actual prediction problems. If the time series problem we predict (such as predicting heartbeat frequency) is not only related to statistical data, but also related to text (such as physician opinions) and images (such as electrocardiogram), we can use MLP, CNN, bert Wait for the clutter together to build a more powerful model.

▲ Source: https://www.kaggle.com/c/avito-demand-prediction/discussion/59880

Theoretical articles:

[Dry goods] Explain LSTM and its Python code implementation in simple terms :

LSTM principle

https://zhuanlan.zhihu.com/p/104475016

Detailed explanation of the principle of Seq2Seq - early bug - blog garden :

seq2seq principle

https://www.cnblogs.com/liuxiaochong/p/14399416.html

Wavenet principle and implementation :

The principle of wavenet

https://zhuanlan.zhihu.com/p/28849767

How CNN Convolutional Neural Networks handle one-dimensional time series data :

1D-CNN processing time series data

https://www.ai8py.com/cnn-in-keras-for-time-sequences.html

Detailed explanation of Transformer for TimeSeries time series prediction algorithm :

transformer time series prediction

https://zhuanlan.zhihu.com/p/391337035

Practical articles:

Python implementation of seq2seq model - natural language processing application :

seq2seq model implementation

https://dataxujing.github.io/seq2seqlearn/chapter3/

machinelearningmastery.com：

LSTM in practice

https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-pyth

Conv1d-WaveNet-Forecast Stock price：

A wavenet model predicts stock prices

https://www.kaggle.com/bhavinmoriya/conv1d-wavenet-forecast-stock-price

towardsdatascience.com/：

transformer time series prediction data

https://towardsdatascience.com/how-to-use-transformer-networks-to-build-a-forecasting-model-297f9270e630

Keras documentation :

Timeseries classification with a Transformer model: Transformer handles time series data classification

https://keras.io/examples/timeseries/timeseries_transformer_classification/

kaggle.com/fatmakursun/：

CNN prediction model

https://www.kaggle.com/fatmakursun/predict-sales-time-series-with-cnn

Summary: The deep learning model can solve basically all timing problems, and the model can automatically learn feature engineering, which greatly reduces labor; however, it requires high model architecture capabilities.

Finally, I will attach some more classic data mining competition links and solutions. If you can understand the data and code, you will definitely benefit a lot. If you are very interested in a certain game solution, I will explain it in detail later.

1) Prediction of website traffic:

RNN seq2seq model:

https://github.com/Arturus/kaggle-web-traffic

xgboost and MLP models:

https://github.com/jfpuget/Kaggle/tree/master/WebTrafficPrediction

Kalman filter:

https://github.com/oseiskar/simdkalman

CNN model:

https://github.com/sjvasquez/web-traffic-forecasting

2) Prediction of restaurant customer volume

Feature Engineering + lgb:

https://www.kaggle.com/plantsgo/solution-public-0-471-private-0-505

Feature Engineering + lgb:

https://www.kaggle.com/pureheart/1st-place-lgb-model-public-0-470-private-0-502

3) Open channel prediction

wavenet model:

https://www.kaggle.com/vicensgaitan/2-wavenet-swa

1D-CNN model:

https://www.kaggle.com/kmat2019/u-net-1d-cnn-with-keras

seq2seq model:

https://www.kaggle.com/brandenkmurray/seq2seq-rnn-with-gru

4) Pulmonary pressure prediction

transformer model:

https://www.kaggle.com/cdeotte/tensorflow-transformer-0-112

Bidirectional lstm model:

https://www.kaggle.com/tenffe/finetune-of-tensorflow-bidirectional-lstm

The time series problem is broad and profound, and the application scenarios are very wide. In fact, many forecasting problems can be regarded as time series problems, such as stock/futures/foreign exchange price forecasting, website/restaurant/hotel/traffic flow forecasting, store commodity inventory/sales forecasting and so on. Once you master time series forecasting, you may hold a key to the future.

Summary of Time Series Forecasting Methods: From Theory to Practice (with Kaggle Classic Competition Scheme)

Guess you like