Abstract:The Controllable Variational Autoencoder (ControlVAE) combines automatic control theory with the basic VAE model to manipulate the KL-divergence for overcoming posterior collapse and learning disentangled representations. It has shown success in a variety of applications, such as image generation, disentangled representation learning, and language modeling. However, when it comes to disentangled representation learning, ControlVAE does not delve into the rationale behind it. The goal of this paper is to develop a deeper understanding of ControlVAE in learning disentangled representations, including the choice of a desired KL-divergence (i.e, set point), and its stability during training. We first fundamentally explain its ability to disentangle latent variables from an information bottleneck perspective. We show that KL-divergence is an upper bound of the variational information bottleneck. By controlling the KL-divergence gradually from a small value to a target value, ControlVAE can disentangle the latent factors one by one. Based on this finding, we propose a new DynamicVAE that leverages a modified incremental PI (proportionalintegral) controller, a variant of the proportional-integralderivative (PID) algorithm, and employs a moving average as well as a hybrid annealing method to evolve the value of KL-divergence smoothly in a tightly controlled fashion. In addition, we analytically derive a lower bound of the set point for disentangling. We then theoretically prove the stability of the proposed approach. Evaluation results on multiple benchmark datasets demonstrate that DynamicVAE achieves a good trade-off between the disentanglement and reconstruction quality. We also discover that it can separate disentangled representation learning and re-construction via manipulating the desired KL-divergence.

Variational Information Bottleneck on Vector Quantized Autoencoders

Uncertainty in the Variational Information Bottleneck

Deep Variational Multivariate Information Bottleneck -- A Framework for Variational Losses

Bi-fidelity variational auto-encoder for uncertainty quantification

Tighter Bounds on the Information Bottleneck with Application to Deep Learning

Disentangled Variational Information Bottleneck for Multiview Representation Learning

A Variational AutoEncoder for Transformers with Nonparametric Variational Information Bottleneck

Variational Predictive Information Bottleneck

Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training

SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization

Variational Bayesian Quantization

On the Difference Between the Information Bottleneck and the Deep Information Bottleneck

PRI-VAE: Principle-of-Relevant-Information Variational Autoencoders

Hidden Talents of the Variational Autoencoder

Visualizing Information Bottleneck through Variational Inference

Rethinking Controllable Variational Autoencoders

Information Constraints on Auto-Encoding Variational Bayes

Three Variations on Variational Autoencoders

Quantization-Based Regularization for Autoencoders

Gaussian Mixture Vector Quantization with Aggregated Categorical Posterior