Awesome Self-Supervised Learning for Time Series (SSL4TS)

Stars

A professionally curated list of awesome resources (paper, code, data, etc.) on Self-Supervised Learning for Time Series (SSL4TS), which is the first work to comprehensively and systematically summarize the recent advances of Self-Supervised Learning for modeling time series data to the best of our knowledge.

We will continue to update this list with the newest resources. If you find any missed resources (paper/code) or errors, please feel free to open an issue or make a pull request.

For general AI for Time Series (AI4TS) Papers, Tutorials, and Surveys at the Top AI Conferences and Journals, please check This Repo.

Survey Paper (IEEE TPAMI 2024)

Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects

Kexin Zhang, Qingsong Wen, Chaoli Zhang, Rongyao Cai, Ming Jin, Yong Liu, James Zhang, Yuxuan Liang, Guansong Pang, Dongjin Song, Shirui Pan.

If you find this repository helpful for your work, please kindly cite our TPAMI'24 paper.

@article{zhang2024ssl4ts,
  title={Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects},
  author={Kexin Zhang and Qingsong Wen and Chaoli Zhang and Rongyao Cai and Ming Jin and Yong Liu and James Zhang and Yuxuan Liang and Guansong Pang and Dongjin Song and Shirui Pan},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
  year={2024}
}

Taxonomy of Self-Supervised Learning for Time Series

Category of Self-Supervised Learning for Time Series

Generative-based Methods on SSL4TS

In this category, the pretext task is to generate the expected data based on a given view of the data. In the context of time series modeling, the commonly used pretext tasks include using the past series to forecast the future windows or specific time stamps, using the encoder and decoder to reconstruct the input, and forecasting the unseen part of the masked time series. This section sorts out the existing self-supervised representation learning methods in time series modeling from the perspectives of autoregressive-based forecasting, autoencoder-based reconstruction, and diffusion-based generation. It should be noted that autoencoder-based reconstruction task is also viewed as an unsupervised framework. In the context of SSL, we mainly use the reconstruction task as a pretext task, and the final goal is to obtain the representations through autoencoder models. The illustration of the generative-based SSL for time series is shown in Fig. 3.

Autoregressive-based forecasting

Timeseries anomaly detection using temporal hierarchical one-class network, in NeurIPS, 2020. [paper]
Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series, in ACM Transactions on Knowledge Discovery from Data, 2022. [paper]
Graph neural network-based anomaly detection in multivariate time series, in AAAI, 2021. [paper]
Semisupervised time series classification model with self-supervised learning, in Engineering Applications of Artificial Intelligence, 2022. [paper]

Autoencoder-based reconstruction

TimeNet: Pre-trained deep recurrent neural network for time series classification, in arXiv, 2017. [paper]
Unsupervised pre-training of a deep LSTM-based stacked autoencoder for multivariate time series forecasting problems, in Scientific Reports, 2019. [paper]
Autowarp: Learning a warping distance from unlabeled time series using sequence autoencoders, in NeurIPS, 2018. [paper]
Practical approach to asynchronous multivariate time series anomaly detection and localization, in KDD, 2021. [paper]
Learning representations for time series clustering, in NeurIPS, 2019. [paper]
USAD: Unsupervised anomaly detection on multivariate time series, in KDD, 2020 [paper]
Learning sparse latent graph representations for anomaly detection in multivariate time series, in KDD, 2022. [paper]
Wind turbine fault detection using a denoising autoencoder with temporal information, in IEEE/ASME Transactions on Mechatronics, 2018 [paper]
Denoising temporal convolutional recurrent autoencoders for time series classification, in Information Sciences, 2022. [paper]
Pre-training enhanced spatial-temporal graph neural network for multivariate time series forecasting, in KDD, 2022. [paper]
A transformer-based framework for multivariate time series representation learning, in KDD, 2021. [paper]
Multi-variate time series forecasting on variable subsets, in KDD, 2022. [paper]
TARNet: Task-aware reconstruction for time-series transformer, in KDD, 2022. [paper]
Learning latent seasonal-trend representations for time series forecasting, in NeurIPS, 2022. [paper] [repo]
Multivariate time series anomaly detection and interpretation using hierarchical inter-metric and temporal embedding, in KDD, 2021. [paper]
Robust anomaly detection for multivariate time series through stochastic recurrent neural network, in KDD, 2019. [paper]
GRELEN: Multivariate time series anomaly detection from the perspective of graph relational learning, in IJCAI, 2022. [paper]
Deep variational graph convolutional recurrent network for multivariate time series anomaly detection, in ICML, 2022. [paper]
Heteroscedastic temporal variational autoencoder for irregularly sampled time series, in ICLR, 2022. [paper]
Learning from irregularly-sampled time series: A missing data perspective, in ICML, 2020. [paper]
TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders, in arXiv, 2023. [paper] [code]

Diffusion-based generation

CSDI: Conditional score-based diffusion models for probabilistic time series imputation, in NeurIPS, 2021. [paper]
Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting, in ICML, 2021. [paper]
Generative time series forecasting with diffusion, denoise, and disentanglement, in NeurIPS, 2022. [paper]
ImDiffusion: Imputed diffusion models for multivariate time series anomaly detection, in arXiv, 2023. [paper]
Diffusion-based time series imputation and forecasting with structured state space models, in Transactions on Machine Learning Research, 2022. [paper]
Diffload: Uncertainty quantification in load forecasting with diffusion model, in arXiv, 2023. [paper]
DiffSTG: Probabilistic spatio-temporal graph forecasting with denoising diffusion models, in SIGSPATIAL, 2023. [paper]
PriSTI: A Conditional Diffusion Framework for Spatiotemporal Imputation, in ICDE, 2024. [paper] [code]

Contrastive-based Methods on SSL4TS

Contrastive learning is a widely used self-supervised learning strategy, showing a strong learning ability in computer vision and natural language processing. Unlike discriminative models that learn a mapping rule to true labels and generative models that try to reconstruct inputs, contrastive-based methods aim to learn data representations by contrasting between positive and negative samples. Specifically, positive samples should have similar representations, while negative samples have different representations. Therefore, the selection of positive samples and negative samples is very important to contrastive-based methods. This section sorts out and summarizes the existing contrastive-based methods in time series modeling according to the selection of positive and negative samples. The illustration of the contrastive-based SSL for time series is shown in Fig. 4.

Sampling contrast

Unsupervised scalable representation learning for multivariate time series, in NeurIPS, 2019. [paper]
Unsupervised representation learning for time series with temporal neighborhood coding, in ICLR, 2021. [paper]
Neighborhood contrastive learning applied to online patient monitoring, in ICML, 2021. [paper]

Prediction contrast

Representation learning with contrastive predictive coding, in arXiv, 2018. [paper]
Detecting anomalies within time series using local neural transformations, in arXiv, 2022. [paper]
Contrastive predictive coding for anomaly detection in multi-variate time series data, in arXiv, 2022. [paper]
Time series change point detection with self-supervised contrastive predictive coding, in WWW, 2021. [paper]
Time Series Anomaly Detection using Skip-Step Contrastive Predictive Coding, in NeurIPS Workshop: Self-Supervised Learning-Theory and Practice, 2022. [paper]
Stock trend prediction with multi-granularity data: A contrastive learning approach with adaptive fusion, in CIKM, 2021. [paper]
Time-series representation learning via temporal and contextual contrasting, in IJCAI, 2021. [paper]
Self-supervised contrastive representation learning for semi-supervised time-series classification, in arXiv, 2022. [paper]

Augmentation contrast

TS2Vec: Towards universal representation of time series, in AAAI, 2022. [paper]
CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting, in ICLR, 2022. [paper]
Unsupervised time-series representation learning with iterative bilinear temporal-spectral fusion, in ICML, 2022. [paper]
Self-supervised contrastive pre-training for time series via time-frequency consistency, in NeurIPS, 2022. [paper]
Timeclr: A self-supervised contrastive learning framework for univariate time series representation, in Knowledge-Based Systems, 2022. [paper]
Clocs: Contrastive learning of cardiac signals across space, time, and patients, in ICML, 2021. [paper]
Contrastive learning for unsupervised domain adaptation of time series, in arXiv, 2022. [paper]
Valve Stiction Detection Using Multitimescale Feature Consistent Constraint for Time-Series Data, in IEEE/ASME Transactions on Mechatronics, 2022. [paper]
Multi-Granularity Residual Learning with Confidence Estimation for Time Series Prediction, in WWW, 2022. [paper]
Stock trend prediction with multi-granularity data: A contrastive learning approach with adaptive fusion, in CIKM, 2021. [paper]
Self-supervised