Abstract:Deep learning has revolutionized the last decade, being at the forefront of extraordinary advances in a wide range of tasks including computer vision, natural language processing, and reinforcement learning, to name but a few. However, it is well-known that deep models trained via maximum likelihood estimation tend to be overconfident and give poorly-calibrated predictions. Bayesian deep learning attempts to address this by placing priors on the model parameters, which are then combined with a likelihood to perform posterior inference. Unfortunately, for deep models, the true posterior is intractable, forcing the user to resort to approximations. In this thesis, we explore the use of variational inference (VI) as an approximation, as it is unique in simultaneously approximating the posterior and providing a lower bound to the marginal likelihood. If tight enough, this lower bound can be used to optimize hyperparameters and to facilitate model selection. However, this capacity has rarely been used to its full extent for Bayesian neural networks, likely because the approximate posteriors typically used in practice can lack the flexibility to effectively bound the marginal likelihood. We therefore explore three aspects of Bayesian learning for deep models: 1) we ask whether it is necessary to perform inference over as many parameters as possible, or whether it is reasonable to treat many of them as optimizable hyperparameters; 2) we propose a variational posterior that provides a unified view of inference in Bayesian neural networks and deep Gaussian processes; 3) we demonstrate how VI can be improved in certain deep Gaussian process models by analytically removing symmetries from the posterior, and performing inference on Gram matrices instead of features. We hope that our contributions will provide a stepping stone to fully realize the promises of VI in the future.

Variational Inference For Nonparametric Subspace Dictionary Learning With Hierarchical Beta Process

A Variational Bayesian Dictionary Learning for Process Monitoring

A Hierarchical Variational Bayesian Approximation Approach in Acoustic Imaging

Sparse Representation over Overcomplete Dictionary Based on Bayesian Nonparametric Approximation

Nonparametric tensor dictionary learning with beta process priors.

A Variational Bayesian Approximation Approach Via A Sparsity Enforcing Prior In Acoustic Imaging

A Nonparametric Bayesian Approach To Joint Multiple Dictionary Learning With Separate Image Sources

Sparse Bayesian Dictionary Learning with a Gaussian Hierarchical Model.

Stick-Breaking Dependent Beta Processes with Variational Inference

Orthogonality-Promoting Dictionary Learning Via Bayesian Inference.

Non-parametric Bayesian Dictionary Learning Based on Laplace Noise

Extended Variational Inference for Dirichlet Process Mixture of Beta‐Liouville Distributions for Proportional Data Modeling

Variational Bayesian Learning for Dirichlet Process Mixture of Inverted Dirichlet Distributions in Non-Gaussian Image Feature Modeling

Bayesian nonparametric dictionary learning for compressed sensing MRI.

Variational methods for the Dirichlet process

Nonlocal Structured Nonparametric Bayesian Dictionary Learning for Image Denoising

Learning Invariant Subspaces of Koopman Operators--Part 2: Heterogeneous Dictionary Mixing to Approximate Subspace Invariance

Towards Improved Variational Inference for Deep Bayesian Models

Variational inference for Dirichlet process mixtures

A variational autoencoder-based nonnegative matrix factorisation model for deep dictionary learning

Structured Black Box Variational Inference for Latent Time Series Models