Abstract:Time series clustering is fundamental in data analysis for discovering temporal patterns. Despite recent advancements, learning cluster-friendly representations is still challenging, particularly with long and complex time series. Deep temporal clustering methods have been trying to integrate the canonical k-means into end-to-end training of neural networks but fall back on surrogate losses due to the non-differentiability of the hard cluster assignment, yielding sub-optimal solutions. In addition, the autoregressive strategy used in the state-of-the-art RNNs is subject to error accumulation and slow training, while recent research findings have revealed that Transformers are less effective due to time points lacking semantic meaning, to the permutation invariance of attention that discards the chronological order and high computation cost. In light of these observations, we present LoSTer which is a novel dense autoencoder architecture for the long-sequence time series clustering problem (LSTC) capable of optimizing the k-means objective via the Gumbel-softmax reparameterization trick and designed specifically for accurate and fast clustering of long time series. Extensive experiments on numerous benchmark datasets and two real-world applications prove the effectiveness of LoSTer over state-of-the-art RNNs and Transformer-based deep clustering methods.

What problem does this paper attempt to address?

This paper focuses on the problem of Long-Sequence Time Series Clustering (LSTC). Current methods face challenges when dealing with high-dimensional and large-scale datasets, particularly when the time series are complex and long. Traditional k-means algorithm performs poorly on nonlinear dynamical time series, while deep learning methods such as RNNs and Transformers also have limitations such as error accumulation, slow training speed, and insufficient capturing ability of long time series. The paper proposes a novel dense autoencoder architecture called LoSTer, specifically designed for the LSTC problem. It can optimize the k-means objective function and solve the non-differentiability issue of hard clustering assignments through the Gumbel-softmax reparameterization technique. Unlike existing methods, LoSTer does not require a predetermined number of iterations for cluster assignment, avoiding inconsistent assignments and providing more accurate clustering. LoSTer compares with deep clustering methods based on RNNs and Transformers. Experimental results show that LoSTer demonstrates superior performance in both clustering accuracy and training speed on multiple benchmark datasets as well as two real-world applications. Additionally, LoSTer overcomes the recurrent error accumulation of RNNs and the limitation of attention mechanism in Transformers on long time series by adopting a simple dense structure, while reducing computational cost. In summary, the main contributions of the paper include: 1. Introducing the first end-to-end deep learning model, LoSTer, which directly optimizes the k-means objective without using suboptimal alternative losses. 2. Elaborating the limitations of Transformers in long sequence time series clustering. 3. Designing a novel dense autoencoder specifically for LSTC, effectively addressing the limitations of RNNs and Transformers. 4. LoSTer achieves significantly faster training speed than RNNs and Transformer baseline models in large-scale LSTC settings.

Concrete Dense Network for Long-Sequence Time Series Clustering

End-to-end deep representation learning for time series clustering: a comparative study

Deep Temporal Contrastive Clustering

Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks

Recurrent Deep Divergence-based Clustering for simultaneous feature learning and clustering of variable length time series

Design of Hierarchical Neural Networks Using Deep LSTM and Self-organizing Dynamical Fuzzy-Neural Network Architecture

Deep Temporal Clustering : Fully Unsupervised Learning of Time-Domain Features

Deep Learning with Long Short-Term Memory for Time Series Prediction

A TCN-Linear Hybrid Model for Chaotic Time Series Forecasting

Deep Spatiotemporal Clustering: A Temporal Clustering Approach for Multi-dimensional Climate Data

Hierarchical Clustering using Auto-encoded Compact Representation for Time-series Analysis

Forecasting Across Time Series Databases using Recurrent Neural Networks on Groups of Similar Series: A Clustering Approach

Dynamic Multi-Network Mining of Tensor Time Series

A Denoising Time Window Algorithm for Optimizing LSTM Prediction

Deep Fusion Clustering Network With Reliable Structure Preservation

TSLANet: Rethinking Transformers for Time Series Representation Learning

Randomnet: clustering time series using untrained deep neural networks

Improving long-term multivariate time series forecasting with a seasonal-trend decomposition-based 2-dimensional temporal convolution dense network

RWKV-TS: Beyond Traditional Recurrent Neural Network for Time Series Tasks

Deep Discriminative Latent Space for Clustering