Abstract:Deep neural networks (DNNs) suffer from the spectral bias, wherein DNNs typically exhibit a tendency to prioritize the learning of lower-frequency components of a function, struggling to capture its high-frequency features. This paper is to address this issue. Notice that a function having only low frequency components may be well-represented by a shallow neural network (SNN), a network having only a few layers. By observing that composition of low frequency functions can effectively approximate a high-frequency function, we propose to learn a function containing high-frequency components by composing several SNNs, each of which learns certain low-frequency information from the given data. We implement the proposed idea by exploiting the multi-grade deep learning (MGDL) model, a recently introduced model that trains a DNN incrementally, grade by grade, a current grade learning from the residue of the previous grade only an SNN composed with the SNNs trained in the preceding grades as features. We apply MGDL to synthetic, manifold, colored images, and MNIST datasets, all characterized by presence of high-frequency features. Our study reveals that MGDL excels at representing functions containing high-frequency information. Specifically, the neural networks learned in each grade adeptly capture some low-frequency information, allowing their compositions with SNNs learned in the previous grades effectively representing the high-frequency features. Our experimental results underscore the efficacy of MGDL in addressing the spectral bias inherent in DNNs. By leveraging MGDL, we offer insights into overcoming spectral bias limitation of DNNs, thereby enhancing the performance and applicability of deep learning models in tasks requiring the representation of high-frequency information. This study confirms that the proposed method offers a promising solution to address the spectral bias of DNNs.

Frequency Principle in deep learning: an overview

Overview frequency principle/spectral bias in deep learning

Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

Frequency Principle in Deep Learning Beyond Gradient-descent-based Training

Theory of the Frequency Principle for General Deep Neural Networks

Training Behavior of Deep Neural Network in Frequency Domain.

Deep Frequency Principle Towards Understanding Why Deeper Learning Is Faster

Frequency Principle in Deep Learning with General Loss Functions and Its Potential Application.

On the exact computation of linear frequency principle dynamics and its generalization

Interpretations of Frequency Domain Analyses of Neural Entrainment: Periodicity, Fundamental Frequency, and Harmonics.

Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks.

Frequency Principle in Broad Learning System

Frequency principle for quantum machine learning via Fourier analysis

Fourier Subspace-Based Deep Learning Method for Inverse Design of Frequency Selective Surface

An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network

A Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks

Understanding the dynamics of the frequency bias in neural networks

Understanding Training and Generalization in Deep Learning by Fourier Analysis.

Going Deeper in Frequency Convolutional Neural Network: A Theoretical Perspective

Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning

Deep learning in spectral analysis: Modeling and imaging