Abstract:High-dimensional time series are common in many domains. Since human cognition is not optimized to work well in high-dimensional spaces, these areas could benefit from interpretable low-dimensional representations. However, most representation learning algorithms for time series data are difficult to interpret. This is due to non-intuitive mappings from data features to salient properties of the representation and non-smoothness over time. To address this problem, we propose a new representation learning framework building on ideas from interpretable discrete dimensionality reduction and deep generative modeling. This framework allows us to learn discrete representations of time series, which give rise to smooth and interpretable embeddings with superior clustering performance. We introduce a new way to overcome the non-differentiability in discrete representation learning and present a gradient-based version of the traditional self-organizing map algorithm that is more performant than the original. Furthermore, to allow for a probabilistic interpretation of our method, we integrate a Markov model in the representation space. This model uncovers the temporal transition structure, improves clustering performance even further and provides additional explanatory insights as well as a natural representation of uncertainty. We evaluate our model in terms of clustering performance and interpretability on static (Fashion-)MNIST data, a time series of linearly interpolated (Fashion-)MNIST images, a chaotic Lorenz attractor system with two macro states, as well as on a challenging real world medical time series application on the eICU data set. Our learned representations compare favorably with competitor methods and facilitate downstream tasks on the real world data.

Learning multilevel distributed representations for high-dimensional sequences

Learning nonlinear manifolds based on mixtures of localized linear manifolds under a self-organizing framework

Modeling sequential data using higher-order relational features and predictive training

High-dimensional learning of narrow neural networks

A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning

Learning Sequential Latent Variable Models from Multimodal Time Series Data

Flexible and accurate inference and learning for deep generative models

Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues

Disentangled Dynamic Representations from Unordered Data

Learning Latent Process from High-Dimensional Event Sequences Via Efficient Sampling

Hyperdimensional Vector Tsetlin Machines with Applications to Sequence Learning and Generation

Unsupervised Learning of Video Representations using LSTMs

Nonlinear time-series embedding by monotone variational inequality

Learning Sparse, Distributed Representations using the Hebbian Principle

Next state prediction gives rise to entangled, yet compositional representations of objects

SOM-VAE: Interpretable Discrete Representation Learning on Time Series

Learning a Generative Motion Model from Image Sequences based on a Latent Motion Matrix

Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokens

Learning nonlinear dynamic models

DenseHMM: Learning Hidden Markov Models by Learning Dense Representations

A Novel Framework for Learning Stochastic Representations for Sequence Generation and Recognition