Abstract:In this work, we investigate a particular implicit bias in the gradient descent training process, which we term "Feature Averaging", and argue that it is one of the principal factors contributing to non-robustness of deep neural networks. Despite the existence of multiple discriminative features capable of classifying data, neural networks trained by gradient descent exhibit a tendency to learn the average (or certain combination) of these features, rather than distinguishing and leveraging each feature individually. In particular, we provide a detailed theoretical analysis of the training dynamics of gradient descent in a two-layer ReLU network for a binary classification task, where the data distribution consists of multiple clusters with orthogonal cluster center vectors. We rigorously prove that gradient descent converges to the regime of feature averaging, wherein the weights associated with each hidden-layer neuron represent an average of the cluster centers (each center corresponding to a distinct feature). It leads the network classifier to be non-robust due to an attack that aligns with the negative direction of the averaged features. Furthermore, we prove that, with the provision of more granular supervised information, a two-layer multi-class neural network is capable of learning individual features, from which one can derive a binary classifier with the optimal robustness under our setting. Besides, we also conduct extensive experiments using synthetic datasets, MNIST and CIFAR-10 to substantiate the phenomenon of feature averaging and its role in adversarial robustness of neural networks. We hope the theoretical and empirical insights can provide a deeper understanding of the impact of the gradient descent training on feature learning process, which in turn influences the robustness of the network, and how more detailed supervision may enhance model robustness.

Hierarchical Simplicity Bias of Neural Networks

Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data

Identifying Spurious Biases Early in Training through the Lens of Simplicity Bias

Simplicity Bias via Global Convergence of Sharpness Minimization

Simplicity bias and optimization threshold in two-layer ReLU networks

A distributional simplicity bias in the learning dynamics of transformers

Neural Networks Learn Statistics of Increasing Complexity

Complexity Matters: Dynamics of Feature Learning in the Presence of Spurious Correlations

Synthetic Simplicity: Unveiling Bias in Medical Data Augmentation

Overcoming Simplicity Bias in Deep Networks using a Feature Sieve

Dynamic Hierarchical Mimicking Towards Consistent Optimization Objectives

Impact of Label Noise on Learning Complex Features

Hierarchical interpretations for neural network predictions

Understanding Simplicity Bias towards Compositional Mappings via Learning Dynamics

Learning Hierarchical Visual Representations in Deep Neural Networks Using Hierarchical Linguistic Labels

Learned feature representations are biased by complexity, learning order, position, and more

Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networks

Learning Inductive Biases with Simple Neural Networks

The Persistence of Neural Collapse Despite Low-Rank Bias: An Analytic Perspective Through Unconstrained Features

Understanding Visual Feature Reliance through the Lens of Complexity