Abstract:The Controllable Variational Autoencoder (ControlVAE) combines automatic control theory with the basic VAE model to manipulate the KL-divergence for overcoming posterior collapse and learning disentangled representations. It has shown success in a variety of applications, such as image generation, disentangled representation learning, and language modeling. However, when it comes to disentangled representation learning, ControlVAE does not delve into the rationale behind it. The goal of this paper is to develop a deeper understanding of ControlVAE in learning disentangled representations, including the choice of a desired KL-divergence (i.e, set point), and its stability during training. We first fundamentally explain its ability to disentangle latent variables from an information bottleneck perspective. We show that KL-divergence is an upper bound of the variational information bottleneck. By controlling the KL-divergence gradually from a small value to a target value, ControlVAE can disentangle the latent factors one by one. Based on this finding, we propose a new DynamicVAE that leverages a modified incremental PI (proportionalintegral) controller, a variant of the proportional-integralderivative (PID) algorithm, and employs a moving average as well as a hybrid annealing method to evolve the value of KL-divergence smoothly in a tightly controlled fashion. In addition, we analytically derive a lower bound of the set point for disentangling. We then theoretically prove the stability of the proposed approach. Evaluation results on multiple benchmark datasets demonstrate that DynamicVAE achieves a good trade-off between the disentanglement and reconstruction quality. We also discover that it can separate disentangled representation learning and re-construction via manipulating the desired KL-divergence.

CauF-VAE: Causal Disentangled Representation Learning with VAE and Causal Flows

Causal Flow-based Variational Auto-Encoder for Disentangled Causal Representation Learning

Learning Causally Disentangled Representations via the Principle of Independent Causal Mechanisms

SCM-VAE: Learning Identifiable Causal Representations Via Structural Knowledge.

Concept-free Causal Disentanglement with Variational Graph Auto-Encoder

Causal Representation Learning Via Counterfactual Intervention

Disentangled Representation Learning for Causal Inference with Instruments

Rethinking Controllable Variational Autoencoders

Conditional Flow Variational Autoencoders for Structured Sequence Prediction

Causal Effect Estimation with Variational AutoEncoder and the Front Door Criterion

f-VAEs: Improve VAEs with Conditional Flows

Deep Causal Generative Models with Property Control

Causal disentanglement of multimodal data

Causal Inference with Conditional Front-Door Adjustment and Identifiable Variational Autoencoder

Guided Variational Autoencoder for Disentanglement Learning

Diffusion-Based Causal Representation Learning

Causal Diffusion Autoencoders: Toward Counterfactual Generation via Diffusion Probabilistic Models

Tions via invertible generative flows

De-Biasing Generative Models using Counterfactual Methods

Learning Conditional Instrumental Variable Representation for Causal Effect Estimation

Causal Recurrent Variational Autoencoder for Medical Time Series Generation