Abstract:Variational dropout (VD) is a generalization of Gaussian dropout, which aims at inferring the posterior of network weights based on a log-uniform prior on them to learn these weights as well as dropout rate simultaneously. The log-uniform prior not only interprets the regularization capacity of Gaussian dropout in network training, but also underpins the inference of such posterior. However, the log-uniform prior is an improper prior (i.e., its integral is infinite), which causes the inference of posterior to be ill-posed, thus restricting the regularization performance of VD. To address this problem, we present a new generalization of Gaussian dropout, termed variational Bayesian dropout (VBD), which turns to exploit a hierarchical prior on the network weights and infer a new joint posterior. Specifically, we implement the hierarchical prior as a zero-mean Gaussian distribution with variance sampled from a uniform hyper-prior. Then, we incorporate such a prior into inferring the joint posterior over network weights and the variance in the hierarchical prior, with which both the network training and dropout rate estimation can be cast into a joint optimization problem. More importantly, the hierarchical prior is a proper prior which enables the inference of posterior to be well-posed. In addition, we further show that the proposed VBD can be seamlessly applied to network compression. Experiments on classification and network compression demonstrate the superior performance of the proposed VBD in regularizing network training.

Variable Selection Using Deep Variational Information Bottleneck with Drop-Out-One Loss

Exploring the Trade-Off in the Variational Information Bottleneck for Regression with a Single Training Run

Tighter Bounds on the Information Bottleneck with Application to Deep Learning

Deep Variational Multivariate Information Bottleneck -- A Framework for Variational Losses

Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training

Bayesian variable selection in linear regression models with instrumental variables

Variational Bayesian Dropout with a Hierarchical Prior

On the Difference Between the Information Bottleneck and the Deep Information Bottleneck

Distributed Variational Information Bottleneck for IOT Environments

Cauchy-Schwarz Divergence Information Bottleneck for Regression

A Novel Variational Bayesian Method for Variable Selection in Logistic Regression Models.

Uncertainty in the Variational Information Bottleneck

Variable Importance Assessments and Backward Variable Selection for High-Dimensional Data

Generalized Information Bottleneck for Gaussian Variables

Visualizing Information Bottleneck through Variational Inference

Drop-Bottleneck: Learning Discrete Compressed Representation for Noise-Robust Exploration

Sequential Information Bottleneck Network For Rul Prediction

A Variational Information Bottleneck Method for Network Intrusion Detection

Relevant sparse codes with variational information bottleneck