Denoising-Aware Contrastive Learning for Noisy Time Series

Shuang Zhou,Daochen Zha,Xiao Shen,Xiao Huang,Rui Zhang,Fu-Lai Chung

2024-06-07

Abstract:Time series self-supervised learning (SSL) aims to exploit unlabeled data for pre-training to mitigate the reliance on labels. Despite the great success in recent years, there is limited discussion on the potential noise in the time series, which can severely impair the performance of existing SSL methods. To mitigate the noise, the de facto strategy is to apply conventional denoising methods before model training. However, this pre-processing approach may not fully eliminate the effect of noise in SSL for two reasons: (i) the diverse types of noise in time series make it difficult to automatically determine suitable denoising methods; (ii) noise can be amplified after mapping raw data into latent space. In this paper, we propose denoising-aware contrastive learning (DECL), which uses contrastive learning objectives to mitigate the noise in the representation and automatically selects suitable denoising methods for every sample. Extensive experiments on various datasets verify the effectiveness of our method. The code is open-sourced.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

This paper focuses on the problem of self-supervised learning (SSL) on time series data with noise. Existing time series SSL methods have limited effectiveness in handling noise due to the diverse types of noise and the amplification in the feature space. To address this, the paper proposes a framework called Denoising-aware Contrastive Learning (DECL) to reduce noise in representation learning and automatically select appropriate denoising methods.

In DECL, an autoregressive encoder is first used to generate informative latent representations. Positive samples are created by introducing denoising methods (such as LOESS), while negative samples are created by adding noise. By optimizing the contrastive learning objective, the representations are guided to be close to positive samples and far from negative samples, thus reducing noise. Additionally, the paper proposes an automatic denoiser selection strategy that uses reconstruction error to determine the most suitable denoising method for each sample.

Experiments demonstrate the effectiveness of DECL on various datasets, improving the accuracy of representation learning and exhibiting robustness to different levels of noise. Compared to existing methods, it learns representations with less noise and performs superiorly under both pretraining and fine-tuning settings.

In summary, the paper addresses the problem of effectively reducing noise to improve the quality of representation learning in self-supervised learning on time series data with noise.