Abstract:Variational Autoencoder (VAE) has achieved promising success since its emergence. In recent years, its various variants have been developed, especially those works which extend VAE to handle sequential data [1, 2, 5, 7]. However, these works either do not generate sequential latent variables, or encode latent variables only based on inputs from earlier time-steps. We believe that in real-world situations, encoding latent variables at a specific time-step should be based on not only previous observations, but also succeeding samples. In this work, we emphasize such fact and theoretically derive the bidirectional Long Short-Term Memory Variational Autoencoder (bLSTM-VAE), a novel variant of VAE whose encoders and decoders are implemented by bidirectional Long Short-Term Memory (bLSTM) networks. The proposed bLSTM-VAE can encode sequential inputs as an equal-length sequence of latent variables. A latent variable at a specific time-step is encoded by simultaneously processing observations from the first time-step till current time-step in a forward order and observations from current time-step till the last timestep in a backward order. As a result, we consider that the proposed bLSTM-VAE could learn latent variables reliably by mining the contextual information from the whole input sequence. In order to validate the proposed method, we apply it for gesture recognition using 3D skeletal joint data. The evaluation is conducted on the ChaLearn Look at People gesture dataset and NTU RGB+D dataset. The experimental results show that combining with the proposed bLSTM-VAE, the classification network performs better than when combining with a standard VAE, and also outperforms several state-of-the-art methods.

Joint Training of Variational Auto-Encoder and Latent Energy-Based Model.

Learning Energy-Based Model with Variational Auto-Encoder as Amortized Sampler

Bi-level Doubly Variational Learning for Energy-based Latent Variable Models

Learning Joint Latent Space EBM Prior Model for Multi-layer Generator.

Energy-Calibrated VAE with Test Time Free Lunch

Divergence Triangle for Joint Training of Generator Model, Energy-Based Model, and Inferential Model.

Bidirectional Long Short-Term Memory Variational Autoencoder.

Learning Energy-based Model via Dual-MCMC Teaching

Learning Hierarchical Features with Joint Latent Space Energy-Based Prior

Epanechnikov Variational Autoencoder

How to train your VAE

Learning Latent Space Energy-Based Prior Model

Learning Multimodal Latent Generative Models with Energy-Based Prior

Improving Variational Autoencoders with Density Gap-based Regularization

LDC-VAE: A Latent Distribution Consistency Approach to Variational AutoEncoders

Hidden Talents of the Variational Autoencoder

Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness

Variational Autoencoder With Optimizing Gaussian Mixture Model Priors.

Latent Variable Autoencoder.

eVAE: Evolutionary Variational Autoencoder

Guiding Energy-based Models via Contrastive Latent Variables