Abstract:The Controllable Variational Autoencoder (ControlVAE) combines automatic control theory with the basic VAE model to manipulate the KL-divergence for overcoming posterior collapse and learning disentangled representations. It has shown success in a variety of applications, such as image generation, disentangled representation learning, and language modeling. However, when it comes to disentangled representation learning, ControlVAE does not delve into the rationale behind it. The goal of this paper is to develop a deeper understanding of ControlVAE in learning disentangled representations, including the choice of a desired KL-divergence (i.e, set point), and its stability during training. We first fundamentally explain its ability to disentangle latent variables from an information bottleneck perspective. We show that KL-divergence is an upper bound of the variational information bottleneck. By controlling the KL-divergence gradually from a small value to a target value, ControlVAE can disentangle the latent factors one by one. Based on this finding, we propose a new DynamicVAE that leverages a modified incremental PI (proportionalintegral) controller, a variant of the proportional-integralderivative (PID) algorithm, and employs a moving average as well as a hybrid annealing method to evolve the value of KL-divergence smoothly in a tightly controlled fashion. In addition, we analytically derive a lower bound of the set point for disentangling. We then theoretically prove the stability of the proposed approach. Evaluation results on multiple benchmark datasets demonstrate that DynamicVAE achieves a good trade-off between the disentanglement and reconstruction quality. We also discover that it can separate disentangled representation learning and re-construction via manipulating the desired KL-divergence.

Variational Composite Autoencoders

Asymmetric Variational Autoencoders

From abstract items to latent spaces to observed data and back: Compositional Variational Auto-Encoder

VTAE: Variational Transformer Autoencoder with Manifolds Learning

Hidden Talents of the Variational Autoencoder

On the Encoder-Decoder Incompatibility in Variational Text Modeling and Beyond

Rethinking Controllable Variational Autoencoders

Different Latent Variables Learning In Variational Autoencoder

Fault Detection Based on Variational Autoencoders for Complex Nonlinear Processes

An adaptive dimension reduction algorithm for latent variables of variational autoencoder.

Guided Variational Autoencoder for Disentanglement Learning

A Hierarchical Latent Structure for Variational Conversation Modeling

Stacked Wasserstein Autoencoder

Three Variations on Variational Autoencoders

Neighbor Embedding Variational Autoencoder

An Introduction to Variational Autoencoders

Variational Auto-Decoder: A Method for Neural Generative Modeling from Incomplete Data

Variational Approaches for Auto-Encoding Generative Adversarial Networks

SGVAE: Sequential Graph Variational Autoencoder

Variational Autoencoded Regression: High Dimensional Regression of Visual Data on Complex Manifold

Variational Autoencoders for Efficient Simulation-Based Inference