Abstract:In spite of several notable efforts, explaining the generalization of deterministic non-smooth deep nets, e.g., ReLU-nets, has remained challenging. Existing approaches for deterministic non-smooth deep nets typically need to bound the Lipschitz constant of such deep nets but such bounds are quite large, may even increase with the training set size yielding vacuous generalization bounds. In this paper, we present a new family of de-randomized PAC-Bayes margin bounds for deterministic non-convex and non-smooth predictors, e.g., ReLU-nets. Unlike PAC-Bayes, which applies to Bayesian predictors, the de-randomized bounds apply to deterministic predictors like ReLU-nets. A specific instantiation of the bound depends on a trade-off between the (weighted) distance of the trained weights from the initialization and the effective curvature (`flatness') of the trained predictor. To get to these bounds, we first develop a de-randomization argument for non-convex but smooth predictors, e.g., linear deep networks (LDNs), which connects the performance of the deterministic predictor with a Bayesian predictor. We then consider non-smooth predictors which for any given input realized as a smooth predictor, e.g., ReLU-nets become some LDNs for any given input, but the realized smooth predictors can be different for different inputs. For such non-smooth predictors, we introduce a new PAC-Bayes analysis which takes advantage of the smoothness of the realized predictors, e.g., LDN, for a given input, and avoids dependency on the Lipschitz constant of the non-smooth predictor. After careful de-randomization, we get a bound for the deterministic non-smooth predictor. We also establish non-uniform sample complexity results based on such bounds. Finally, we present extensive empirical results of our bounds over changing training set size and randomness in labels.

PAC-Bayesian Generalization Bounds for MultiLayer Perceptrons

A Probabilistic Representation of Deep Learning for Improving The Information Theoretic Interpretability

Improving Generalization of Complex Models under Unbounded Loss Using PAC-Bayes Bounds

PAC-Bayesian Generalization Bound on Confusion Matrix for Multi-Class Classification

PAC-Bayesian Theory Meets Bayesian Inference

Higher-Order Generalization Bounds: Learning Deep Probabilistic Programs via PAC-Bayes Objectives

PAC-Bayesian Generalization Bounds for Knowledge Graph Representation Learning

A PAC-Bayesian Link Between Generalisation and Flat Minima

Misclassification bounds for PAC-Bayesian sparse deep learning

A General Framework for the Practical Disintegration of PAC-Bayesian Bounds

Leveraging PAC-Bayes Theory and Gibbs Distributions for Generalization Bounds with Complexity Measures

Upper Bound of Bayesian Generalization Error in Partial Concept Bottleneck Model (CBM): Partial CBM outperforms naive CBM

Generalisation bounds for kernel PCA through PAC‐Bayes learning

PAC-Bayes Generalisation Bounds for Dynamical Systems Including Stable RNNs

Bridging the Gap Between Approximation and Learning via Optimal Approximation by ReLU MLPs of Maximal Regularity

Generalization and Estimation Error Bounds for Model-based Neural Networks

Explicitly Bayesian Regularizations in Deep Learning

Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP

De-randomized PAC-Bayes Margin Bounds: Applications to Non-convex and Non-smooth Predictors

Bayes' Power for Explaining In-Context Learning Generalizations

Misclassification excess risk bounds for PAC-Bayesian classification via convexified loss