Abstract:Sparse deep neural networks have proven to be efficient for predictive model building in large-scale studies. Although several works have studied theoretical and numerical properties of sparse neural architectures, they have primarily focused on the edge selection. Sparsity through edge selection might be intuitively appealing; however, it does not necessarily reduce the structural complexity of a network. Instead pruning excessive nodes leads to a structurally sparse network with significant computational speedup during inference. To this end, we propose a Bayesian sparse solution using spike-and-slab Gaussian priors to allow for automatic node selection during training. The use of spike-and-slab prior alleviates the need of an ad-hoc thresholding rule for pruning. In addition, we adopt a variational Bayes approach to circumvent the computational challenges of traditional Markov Chain Monte Carlo (MCMC) implementation. In the context of node selection, we establish the fundamental result of variational posterior consistency together with the characterization of prior parameters. In contrast to the previous works, our theoretical development relaxes the assumptions of the equal number of nodes and uniform bounds on all network weights, thereby accommodating sparse networks with layer-dependent node structures or coefficient bounds. With a layer-wise characterization of prior inclusion probabilities, we discuss the optimal contraction rates of the variational posterior. We empirically demonstrate that our proposed approach outperforms the edge selection method in computational complexity with similar or better predictive performance. Our experimental evidence further substantiates that our theoretical work facilitates layer-wise optimal node recovery.

Neuronized Priors for Bayesian Sparse Linear Regression

Spike-and-slab shrinkage priors for structurally sparse Bayesian neural networks

Bayesian Regression Using a Prior on the Model Fit: The R2-D2 Shrinkage Prior

Sparse estimation of linear model via Bayesian method

Bayesian High-dimensional Linear Regression with Sparse Projection-posterior

Empirical priors for prediction in sparse high-dimensional linear regression

The spike-and-slab lasso and scalable algorithm to accommodate multinomial outcomes in variable selection problems

Bayesian Nonnegative Matrix Factorization with a Truncated Spike-and-Slab Prior

Sparse Linear Regression with Beta Process Priors

Dynamic sparsity on dynamic regression models

Sparse Bayesian Linear Regression Using Generalized Normal Priors.

Disjunct Support Spike and Slab Priors for Variable Selection in Regression under Quasi-sparseness

Layer adaptive node selection in Bayesian neural networks: Statistical guarantees and implementation details

High-dimensional Grouped-regression using Bayesian Sparse Projection-posterior

BNNpriors: A library for Bayesian neural network inference with different prior distributions

Adaptive Bayesian Predictive Inference in High-dimensional Regerssion

Bayesian sparse multiple regression for simultaneous rank reduction and variable selection

Bayesian Joint Spike-and-Slab Graphical Lasso

Bayesian Spike Train Inference via Non-Local Priors

Bayesian Variable Selection for Mixed Effects Model with Shrinkage Prior.

Intuitive Joint Priors for Bayesian Linear Multilevel Models: The R2D2M2 prior