An Introduction to Variational Autoencoders

Diederik P. Kingma,Max Welling

DOI: https://doi.org/10.48550/arXiv.1906.02691

IF: 5.414

2019-06-06

Machine Learning

Abstract:Variational autoencoders provide a principled framework for learning deep latent-variable models and corresponding inference models. In this work, we provide an introduction to variational autoencoders and some important extensions.

What problem does this paper attempt to address?

Hidden Talents of the Variational Autoencoder

Bin Dai,Yu Wang,John Aston,Gang Hua,David Wipf

DOI: https://doi.org/10.48550/arXiv.1706.05148

2019-10-07

Abstract:Variational autoencoders (VAE) represent a popular, flexible form of deep generative model that can be stochastically fit to samples from a given random process using an information-theoretic variational bound on the true underlying distribution. Once so-obtained, the model can be putatively used to generate new samples from this distribution, or to provide a low-dimensional latent representation of existing samples. While quite effective in numerous application domains, certain important mechanisms which govern the behavior of the VAE are obfuscated by the intractable integrals and resulting stochastic approximations involved. Moreover, as a highly non-convex model, it remains unclear exactly how minima of the underlying energy relate to original design purposes. We attempt to better quantify these issues by analyzing a series of tractable special cases of increasing complexity. In doing so, we unveil interesting connections with more traditional dimensionality reduction models, as well as an intrinsic yet underappreciated propensity for robustly dismissing sparse outliers when estimating latent manifolds. With respect to the latter, we demonstrate that the VAE can be viewed as the natural evolution of recent robust PCA models, capable of learning nonlinear manifolds of unknown dimension obscured by gross corruptions.

Machine Learning
Asymmetric Variational Autoencoders

Guoqing Zheng,Yiming Yang,Jaime Carbonell

DOI: https://doi.org/10.48550/arXiv.1711.08352

2018-07-10

Abstract:Variational inference for latent variable models is prevalent in various machine learning problems, typically solved by maximizing the Evidence Lower Bound (ELBO) of the true data likelihood with respect to a variational distribution. However, freely enriching the family of variational distribution is challenging since the ELBO requires variational likelihood evaluations of the latent variables. In this paper, we propose a novel framework to enrich the variational family by incorporating auxiliary variables to the variational family. The resulting inference network doesn't require density evaluations for the auxiliary variables and thus complex implicit densities over the auxiliary variables can be constructed by neural networks. It can be shown that the actual variational posterior of the proposed approach is essentially modeling a rich probabilistic mixture of simple variational posterior indexed by auxiliary variables, thus a flexible inference model can be built. Empirical evaluations on several density estimation tasks demonstrates the effectiveness of the proposed method.

Machine Learning
Latent Variable Modelling Using Variational Autoencoders: A survey

Vasanth Kalingeri

DOI: https://doi.org/10.48550/arXiv.2206.09891

IF: 5.414

2022-06-20

Machine Learning

Abstract:A probability distribution allows practitioners to uncover hidden structure in the data and build models to solve supervised learning problems using limited data. The focus of this report is on Variational autoencoders, a method to learn the probability distribution of large complex datasets. The report provides a theoretical understanding of variational autoencoders and consolidates the current research in the field. The report is divided into multiple chapters, the first chapter introduces the problem, describes variational autoencoders and identifies key research directions in the field. Chapters 2, 3, 4 and 5 dive into the details of each of the key research areas. Chapter 6 concludes the report and suggests directions for future work. A reader who has a basic idea of machine learning but wants to learn about general themes in machine learning research can benefit from the report. The report explains central ideas on learning probability distributions, what people did to make this tractable and goes into details around how deep learning is currently applied. The report also serves a gentle introduction for someone looking to contribute to this sub-field.
Tutorial on Variational Autoencoders

Carl Doersch

DOI: https://doi.org/10.48550/arXiv.1606.05908

IF: 5.414

2016-06-19

Machine Learning

Abstract:In just three years, Variational Autoencoders (VAEs) have emerged as one of the most popular approaches to unsupervised learning of complicated distributions. VAEs are appealing because they are built on top of standard function approximators (neural networks), and can be trained with stochastic gradient descent. VAEs have already shown promise in generating many kinds of complicated data, including handwritten digits, faces, house numbers, CIFAR images, physical models of scenes, segmentation, and predicting the future from static images. This tutorial introduces the intuitions behind VAEs, explains the mathematics behind them, and describes some empirical behavior. No prior knowledge of variational Bayesian methods is assumed.
An Introduction to Autoencoders

Umberto Michelucci

DOI: https://doi.org/10.48550/arXiv.2201.03898

IF: 5.414

2022-01-11

Machine Learning

Abstract:In this article, we will look at autoencoders. This article covers the mathematics and the fundamental concepts of autoencoders. We will discuss what they are, what the limitations are, the typical use cases, and we will look at some examples. We will start with a general introduction to autoencoders, and we will discuss the role of the activation function in the output layer and the loss function. We will then discuss what the reconstruction error is. Finally, we will look at typical applications as dimensionality reduction, classification, denoising, and anomaly detection. This paper contains the notes of a PhD-level lecture on autoencoders given in 2021.
Three Variations on Variational Autoencoders

R. I. Cukier

DOI: https://doi.org/10.48550/arXiv.2212.04451

2023-04-05

Abstract:Variational autoencoders (VAEs) are one class of generative probabilistic latent-variable models designed for inference based on known data. We develop three variations on VAEs by introducing a second parameterized encoder/decoder pair and, for one variation, an additional fixed encoder. The parameters of the encoders/decoders are to be learned with a neural network. The fixed encoder is obtained by probabilistic-PCA. The variations are compared to the Evidence Lower Bound (ELBO) approximation to the original VAE. One variation leads to an Evidence Upper Bound (EUBO) that can be used in conjunction with the original ELBO to interrogate the convergence of the VAE.

Machine Learning
Auto-Encoding Variational Bayes

M. Welling,Diederik P. Kingma

2013-12-20

Abstract:Abstract: How can we perform efficient inference and learning in directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions, and large datasets? We introduce a stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case. Our contributions is two-fold. First, we show that a reparameterization of the variational lower bound yields a lower bound estimator that can be straightforwardly optimized using standard stochastic gradient methods. Second, we show that for i.i.d. datasets with continuous latent variables per datapoint, posterior inference can be made especially efficient by fitting an approximate inference model (also called a recognition model) to the intractable posterior using the proposed lower bound estimator. Theoretical advantages are reflected in experimental results.

Computer Science,Mathematics
Variational Composite Autoencoders

Jiangchao Yao,Ivor Tsang,Ya Zhang

DOI: https://doi.org/10.48550/arXiv.1804.04435

IF: 5.414

2018-04-12

Machine Learning

Abstract:Learning in the latent variable model is challenging in the presence of the complex data structure or the intractable latent variable. Previous variational autoencoders can be low effective due to the straightforward encoder-decoder structure. In this paper, we propose a variational composite autoencoder to sidestep this issue by amortizing on top of the hierarchical latent variable model. The experimental results confirm the advantages of our model.
Variational Autoencoders: A Harmonic Perspective

Alexander Camuto,Matthew Willetts

DOI: https://doi.org/10.48550/arXiv.2105.14866

IF: 5.414

2021-05-31

Machine Learning

Abstract:In this work we study Variational Autoencoders (VAEs) from the perspective of harmonic analysis. By viewing a VAE's latent space as a Gaussian Space, a variety of measure space, we derive a series of results that show that the encoder variance of a VAE controls the frequency content of the functions parameterised by the VAE encoder and decoder neural networks. In particular we demonstrate that larger encoder variances reduce the high frequency content of these functions. Our analysis allows us to show that increasing this variance effectively induces a soft Lipschitz constraint on the decoder network of a VAE, which is a core contributor to the adversarial robustness of VAEs. We further demonstrate that adding Gaussian noise to the input of a VAE allows us to more finely control the frequency content and the Lipschitz constant of the VAE encoder networks. To support our theoretical analysis we run experiments with VAEs with small fully-connected neural networks and with larger convolutional networks, demonstrating empirically that our theory holds for a variety of neural network architectures.
Variational Approaches for Auto-Encoding Generative Adversarial Networks

Mihaela Rosca,Balaji Lakshminarayanan,David Warde-Farley,Shakir Mohamed

DOI: https://doi.org/10.48550/arXiv.1706.04987

IF: 5.414

2017-06-15

Machine Learning

Abstract:Auto-encoding generative adversarial networks (GANs) combine the standard GAN algorithm, which discriminates between real and model-generated data, with a reconstruction loss given by an auto-encoder. Such models aim to prevent mode collapse in the learned generative model by ensuring that it is grounded in all the available training data. In this paper, we develop a principle upon which auto-encoders can be combined with generative adversarial networks by exploiting the hierarchical structure of the generative model. The underlying principle shows that variational inference can be used a basic tool for learning, but with the in- tractable likelihood replaced by a synthetic likelihood, and the unknown posterior distribution replaced by an implicit distribution; both synthetic likelihoods and implicit posterior distributions can be learned using discriminators. This allows us to develop a natural fusion of variational auto-encoders and generative adversarial networks, combining the best of both these methods. We describe a unified objective for optimization, discuss the constraints needed to guide learning, connect to the wide range of existing work, and use a battery of tests to systematically and quantitatively assess the performance of our method.
Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks

Lars Mescheder,Sebastian Nowozin,Andreas Geiger

DOI: https://doi.org/10.48550/arXiv.1701.04722

IF: 5.414

2017-01-17

Machine Learning

Abstract:Variational Autoencoders (VAEs) are expressive latent variable models that can be used to learn complex probability distributions from training data. However, the quality of the resulting model crucially relies on the expressiveness of the inference model. We introduce Adversarial Variational Bayes (AVB), a technique for training Variational Autoencoders with arbitrarily expressive inference models. We achieve this by introducing an auxiliary discriminative network that allows to rephrase the maximum-likelihood-problem as a two-player game, hence establishing a principled connection between VAEs and Generative Adversarial Networks (GANs). We show that in the nonparametric limit our method yields an exact maximum-likelihood assignment for the parameters of the generative model, as well as the exact posterior distribution over the latent variables given an observation. Contrary to competing approaches which combine VAEs with GANs, our approach has a clear theoretical justification, retains most advantages of standard Variational Autoencoders and is easy to implement.
Variational autoencoders in the presence of low-dimensional data: landscape and implicit bias

Frederic Koehler,Viraj Mehta,Chenghui Zhou,Andrej Risteski

DOI: https://doi.org/10.48550/arXiv.2112.06868

2022-05-18

Abstract:Variational Autoencoders are one of the most commonly used generative models, particularly for image data. A prominent difficulty in training VAEs is data that is supported on a lower-dimensional manifold. Recent work by Dai and Wipf (2020) proposes a two-stage training algorithm for VAEs, based on a conjecture that in standard VAE training the generator will converge to a solution with 0 variance which is correctly supported on the ground truth manifold. They gave partial support for that conjecture by showing that some optima of the VAE loss do satisfy this property, but did not analyze the training dynamics. In this paper, we show that for linear encoders/decoders, the conjecture is true-that is the VAE training does recover a generator with support equal to the ground truth manifold-and does so due to an implicit bias of gradient descent rather than merely the VAE loss itself. In the nonlinear case, we show that VAE training frequently learns a higher-dimensional manifold which is a superset of the ground truth manifold.

Machine Learning
A Tutorial on VAEs: From Bayes' Rule to Lossless Compression

Ronald Yu

DOI: https://doi.org/10.48550/arXiv.2006.10273

IF: 5.414

2020-06-18

Machine Learning

Abstract:The Variational Auto-Encoder (VAE) is a simple, efficient, and popular deep maximum likelihood model. Though usage of VAEs is widespread, the derivation of the VAE is not as widely understood. In this tutorial, we will provide an overview of the VAE and a tour through various derivations and interpretations of the VAE objective. From a probabilistic standpoint, we will examine the VAE through the lens of Bayes' Rule, importance sampling, and the change-of-variables formula. From an information theoretic standpoint, we will examine the VAE through the lens of lossless compression and transmission through a noisy channel. We will then identify two common misconceptions over the VAE formulation and their practical consequences. Finally, we will visualize the capabilities and limitations of VAEs using a code example (with an accompanying Jupyter notebook) on toy 2D data.
Ladder Variational Autoencoders

Casper Kaae Sønderby,Tapani Raiko,Lars Maaløe,Søren Kaae Sønderby,Ole Winther

DOI: https://doi.org/10.48550/arXiv.1602.02282

IF: 5.414

2016-02-06

Machine Learning

Abstract:Variational Autoencoders are powerful models for unsupervised learning. However deep models with several layers of dependent stochastic variables are difficult to train which limits the improvements obtained using these highly expressive models. We propose a new inference model, the Ladder Variational Autoencoder, that recursively corrects the generative distribution by a data dependent approximate likelihood in a process resembling the recently proposed Ladder Network. We show that this model provides state of the art predictive log-likelihood and tighter log-likelihood lower bound compared to the purely bottom-up inference in layered Variational Autoencoders and other generative models. We provide a detailed analysis of the learned hierarchical latent representation and show that our new inference model is qualitatively different and utilizes a deeper more distributed hierarchy of latent variables. Finally, we observe that batch normalization and deterministic warm-up (gradually turning on the KL-term) are crucial for training variational models with many stochastic layers.
Connections with Robust PCA and the Role of Emergent Sparsity in Variational Autoencoder Models

Bin Dai,Yu Wang,John Aston,Gang Hua,David Wipf

DOI: https://doi.org/10.5555/3291125.3291166

2018-01-01

Journal of Machine Learning Research

Abstract:Variational autoencoders (VAE) represent a popular, flexible form of deep generative model that can be stochastically fit to samples from a given random process using an information-theoretic variational bound on the true underlying distribution. Once so-obtained, the model can be putatively used to generate new samples from this distribution, or to provide a low-dimensional latent representation of existing samples. While quite effective in numerous application domains, certain important mechanisms which govern the behavior of the VAE are obfuscated by the intractable integrals and resulting stochastic approximations involved. Moreover, as a highly non-convex model, it remains unclear exactly how minima of the underlying energy relate to original design purposes. We attempt to better quantify these issues by analyzing a series of tractable special cases of increasing complexity. In doing so, we unveil interesting connections with more traditional dimensionality reduction models, as well as an intrinsic yet underappreciated propensity for robustly dismissing sparse outliers when estimating latent manifolds. With respect to the latter, we demonstrate that the VAE can be viewed as the natural evolution of recent robust PCA models, capable of learning nonlinear manifolds of unknown dimension obscured by gross corruptions.
Adversarial Attacks on Variational Autoencoders

George Gondim-Ribeiro,Pedro Tabacof,Eduardo Valle

DOI: https://doi.org/10.48550/arXiv.1806.04646

2018-06-13

Abstract:Adversarial attacks are malicious inputs that derail machine-learning models. We propose a scheme to attack autoencoders, as well as a quantitative evaluation framework that correlates well with the qualitative assessment of the attacks. We assess --- with statistically validated experiments --- the resistance to attacks of three variational autoencoders (simple, convolutional, and DRAW) in three datasets (MNIST, SVHN, CelebA), showing that both DRAW's recurrence and attention mechanism lead to better resistance. As autoencoders are proposed for compressing data --- a scenario in which their safety is paramount --- we expect more attention will be given to adversarial attacks on them.

Computer Vision and Pattern Recognition,Machine Learning,Neural and Evolutionary Computing
VTAE: Variational Transformer Autoencoder with Manifolds Learning

Pourya Shamsolmoali,Masoumeh Zareapoor,Huiyu Zhou,Dacheng Tao,Xuelong Li

2023-04-03

Abstract:Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables and these models use a nonlinear function (generator) to map latent samples into the data space. On the other hand, the nonlinearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning. This weak projection, however, can be addressed by a Riemannian metric, and we show that geodesics computation and accurate interpolations between data samples on the Riemannian manifold can substantially improve the performance of deep generative models. In this paper, a Variational spatial-Transformer AutoEncoder (VTAE) is proposed to minimize geodesics on a Riemannian manifold and improve representation learning. In particular, we carefully design the variational autoencoder with an encoded spatial-Transformer to explicitly expand the latent variable model to data on a Riemannian manifold, and obtain global context modelling. Moreover, to have smooth and plausible interpolations while traversing between two different objects' latent representations, we propose a geodesic interpolation network different from the existing models that use linear interpolation with inferior performance. Experiments on benchmarks show that our proposed model can improve predictive accuracy and versatility over a range of computer vision tasks, including image interpolations, and reconstructions.

Computer Vision and Pattern Recognition
From the Expectation Maximisation Algorithm to Autoencoded Variational Bayes

Graham W. Pulford

DOI: https://doi.org/10.48550/arXiv.2010.13551

2021-05-04

Abstract:Although the expectation maximisation (EM) algorithm was introduced in 1970, it remains somewhat inaccessible to machine learning practitioners due to its obscure notation, terse proofs and lack of concrete links to modern machine learning techniques like autoencoded variational Bayes. This has resulted in gaps in the AI literature concerning the meaning of such concepts like "latent variables" and "variational lower bound," which are frequently used but often not clearly explained. The roots of these ideas lie in the EM algorithm. We first give a tutorial presentation of the EM algorithm for estimating the parameters of a $K$-component mixture density. The Gaussian mixture case is presented in detail using $K$-ary scalar hidden (or latent) variables rather than the more traditional binary valued $K$-dimenional vectors. This presentation is motivated by mixture modelling from the target tracking literature. In a similar style to Bishop's 2009 book, we present variational Bayesian inference as a generalised EM algorithm stemming from the variational (or evidential) lower bound, as well as the technique of mean field approximation (or product density transform). We continue the evolution from EM to variational autoencoders, developed by Kingma & Welling in 2014. In so doing, we establish clear links between the EM algorithm and its variational counterparts, hence clarifying the meaning of "latent variables." We provide a detailed coverage of the "reparametrisation trick" and focus on how the AEVB differs from conventional variational Bayesian inference. Throughout the tutorial, consistent notational conventions are used. This unifies the narrative and clarifies the concepts. Some numerical examples are given to further illustrate the algorithms.

Machine Learning
On the Value of Infinite Gradients in Variational Autoencoder Models.

Bin Dai,Li K. Wenliang,David Wipf

2021-01-01

Abstract:A number of recent studies of continuous variational autoencoder (VAE) models have noted, either directly or indirectly, the tendency of various parameter gradients to drift towards infinity during training. Because such gradients could potentially contribute to numerical instabilities, and are often framed as a problematic phenomena to be avoided, it may be tempting to shift to alternative energy functions that guarantee bounded gradients. But it remains an open question: What might the unintended consequences of such a restriction be? To address this issue, we examine how unbounded gradients relate to the regularization of a broad class of autoencoder-based architectures, including VAE models, as applied to data lying on or near a low-dimensional manifold (e.g., natural images). Our main finding is that, if the ultimate goal is to simultaneously avoid over-regularization (high reconstruction errors, sometimes referred to as posterior collapse) and under-regularization (excessive latent dimensions are not pruned from the model), then an autoencoder-based energy function with infinite gradients around optimal representations is provably required per a certain technical sense which we carefully detail. Given that both over- and under-regularization can directly lead to poor generated sample quality or suboptimal feature selection, this result suggests that heuristic modifications to or constraints on the VAE energy function may at times be ill-advised, and large gradients should be accommodated to the extent possible.
Variational Auto-Decoder: A Method for Neural Generative Modeling from Incomplete Data

Amir Zadeh,Yao-Chong Lim,Paul Pu Liang,Louis-Philippe Morency

DOI: https://doi.org/10.48550/arXiv.1903.00840

2021-01-03

Abstract:Learning a generative model from partial data (data with missingness) is a challenging area of machine learning research. We study a specific implementation of the Auto-Encoding Variational Bayes (AEVB) algorithm, named in this paper as a Variational Auto-Decoder (VAD). VAD is a generic framework which uses Variational Bayes and Markov Chain Monte Carlo (MCMC) methods to learn a generative model from partial data. The main distinction between VAD and Variational Auto-Encoder (VAE) is the encoder component, as VAD does not have one. Using a proposed efficient inference method from a multivariate Gaussian approximate posterior, VAD models allow inference to be performed via simple gradient ascent rather than MCMC sampling from a probabilistic decoder. This technique reduces the inference computational cost, allows for using more complex optimization techniques during latent space inference (which are shown to be crucial due to a high degree of freedom in the VAD latent space), and keeps the framework simple to implement. Through extensive experiments over several datasets and different missing ratios, we show that encoders cannot efficiently marginalize the input volatility caused by imputed missing values. We study multimodal datasets in this paper, which is a particular area of impact for VAD models.

Machine Learning,Artificial Intelligence

An Introduction to Variational Autoencoders

Hidden Talents of the Variational Autoencoder

Asymmetric Variational Autoencoders

Latent Variable Modelling Using Variational Autoencoders: A survey

Tutorial on Variational Autoencoders

An Introduction to Autoencoders

Three Variations on Variational Autoencoders

Auto-Encoding Variational Bayes

Variational Composite Autoencoders

Variational Autoencoders: A Harmonic Perspective

Variational Approaches for Auto-Encoding Generative Adversarial Networks

Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks

Variational autoencoders in the presence of low-dimensional data: landscape and implicit bias

A Tutorial on VAEs: From Bayes' Rule to Lossless Compression

Ladder Variational Autoencoders

Connections with Robust PCA and the Role of Emergent Sparsity in Variational Autoencoder Models

Adversarial Attacks on Variational Autoencoders

VTAE: Variational Transformer Autoencoder with Manifolds Learning

From the Expectation Maximisation Algorithm to Autoencoded Variational Bayes

On the Value of Infinite Gradients in Variational Autoencoder Models.

Variational Auto-Decoder: A Method for Neural Generative Modeling from Incomplete Data