Abstract:Bayesian optimization is a highly efficient approach to optimizing objective functions which are expensive to query. These objectives are typically represented by Gaussian process (GP) surrogate models which are easy to optimize and support exact inference. While standard GP surrogates have been well-established in Bayesian optimization, Bayesian neural networks (BNNs) have recently become practical function approximators, with many benefits over standard GPs such as the ability to naturally handle non-stationarity and learn representations for high-dimensional data. In this paper, we study BNNs as alternatives to standard GP surrogates for optimization. We consider a variety of approximate inference procedures for finite-width BNNs, including high-quality Hamiltonian Monte Carlo, low-cost stochastic MCMC, and heuristics such as deep ensembles. We also consider infinite-width BNNs, linearized Laplace approximations, and partially stochastic models such as deep kernel learning. We evaluate this collection of surrogate models on diverse problems with varying dimensionality, number of objectives, non-stationarity, and discrete and continuous inputs. We find: (i) the ranking of methods is highly problem dependent, suggesting the need for tailored inductive biases; (ii) HMC is the most successful approximate inference procedure for fully stochastic BNNs; (iii) full stochasticity may be unnecessary as deep kernel learning is relatively competitive; (iv) deep ensembles perform relatively poorly; (v) infinite-width BNNs are particularly promising, especially in high dimensions.

On the effectiveness of partially deterministic Bayesian neural networks

On Feynman--Kac training of partial Bayesian neural networks

What Are Bayesian Neural Network Posteriors Really Like?

Dangers of Bayesian Model Averaging under Covariate Shift

Single-shot Bayesian approximation for neural networks

BNN-DP: Robustness Certification of Bayesian Neural Networks via Dynamic Programming

Function Space Particle Optimization for Bayesian Neural Networks

Bayesian Neural Networks: A Min-Max Game Framework

Bayesian Inference with Latent Hamiltonian Neural Networks

Improving Bayesian Neural Networks by Adversarial Sampling.

BayesAdapter: Being Bayesian, Inexpensively and Reliably, via Bayesian Fine-tuning

Model Architecture Adaption for Bayesian Neural Networks

Mitigating the Effects of Non-Identifiability on Inference for Bayesian Neural Networks with Latent Variables

Compromise-free Bayesian neural networks

Data Subsampling for Bayesian Neural Networks

Efficient Bayes Inference in Neural Networks through Adaptive Importance Sampling

Do Bayesian Neural Networks Need To Be Fully Stochastic?

Efficient Scaling of Bayesian Neural Networks

A Study of Bayesian Neural Network Surrogates for Bayesian Optimization

Learning Solutions of Stochastic Optimization Problems with Bayesian Neural Networks