Abstract:Training Deep Neural Networks (DNNs) with adversarial examples often results in poor generalization to test-time adversarial data. This paper investigates this issue, known as adversarially robust generalization, through the lens of Rademacher complexity. Building upon the studies by Khim and Loh (2018); Yin et al. (2019), numerous works have been dedicated to this problem, yet achieving a satisfactory bound remains an elusive goal. Existing works on DNNs either apply to a surrogate loss instead of the robust loss or yield bounds that are notably looser compared to their standard counterparts. In the latter case, the bounds have a higher dependency on the width $m$ of the DNNs or the dimension $d$ of the data, with an extra factor of at least $\mathcal{O}(\sqrt{m})$ or $\mathcal{O}(\sqrt{d})$. This paper presents upper bounds for adversarial Rademacher complexity of DNNs that match the best-known upper bounds in standard settings, as established in the work of Bartlett et al. (2017), with the dependency on width and dimension being $\mathcal{O}(\ln(dm))$. The central challenge addressed is calculating the covering number of adversarial function classes. We aim to construct a new cover that possesses two properties: 1) compatibility with adversarial examples, and 2) precision comparable to covers used in standard settings. To this end, we introduce a new variant of covering number called the \emph{uniform covering number}, specifically designed and proven to reconcile these two properties. Consequently, our method effectively bridges the gap between Rademacher complexity in robust and standard generalization.

Researches on Rademacher Complexities in Statistical Learning Theory：A Survey

Generalization Bounds for Stochastic Gradient Langevin Dynamics: A Unified View Via Information Leakage Analysis

Generalization bounds for learning under graph-dependence: A survey

Approximate Inference via Weighted Rademacher Complexity

Permutational Rademacher Complexity: a New Complexity Measure for Transductive Learning

Rademacher complexity analysis for matrixized and vectorized classifier

Rademacher complexity and spin glasses: A link between the replica and statistical theories of learning

Generalization Bounds for Metric and Similarity Learning

Rademacher Complexity for Adversarially Robust Generalization.

Learning from weakly dependent data under Dobrushin's condition

Rademacher complexity for Markov chains : Applications to kernel smoothing and Metropolis-Hasting

Local Rademacher Complexity for Multi-label Learning.

Rademacher Complexity and the Generalization Error of Residual Networks

Bridging the Gap: Rademacher Complexity in Robust and Standard Generalization

Sample Complexity Bounds for Estimating Probability Divergences under Invariances

A Survey on Complexity Measures of Pseudo-Random Sequences

GENERALIZATION BOUNDS OF REGULARIZATION ALGORITHMS DERIVED SIMULTANEOUSLY THROUGH HYPOTHESIS SPACE COMPLEXITY, ALGORITHMIC STABILITY AND DATA QUALITY

Exponential Tail Local Rademacher Complexity Risk Bounds Without the Bernstein Condition

Fast Excess Risk Rates Via Offset Rademacher Complexity.

Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Rényi’s Entropy Perspective