Abstract:Deep neural networks (DNNs) suffer from the spectral bias, wherein DNNs typically exhibit a tendency to prioritize the learning of lower-frequency components of a function, struggling to capture its high-frequency features. This paper is to address this issue. Notice that a function having only low frequency components may be well-represented by a shallow neural network (SNN), a network having only a few layers. By observing that composition of low frequency functions can effectively approximate a high-frequency function, we propose to learn a function containing high-frequency components by composing several SNNs, each of which learns certain low-frequency information from the given data. We implement the proposed idea by exploiting the multi-grade deep learning (MGDL) model, a recently introduced model that trains a DNN incrementally, grade by grade, a current grade learning from the residue of the previous grade only an SNN composed with the SNNs trained in the preceding grades as features. We apply MGDL to synthetic, manifold, colored images, and MNIST datasets, all characterized by presence of high-frequency features. Our study reveals that MGDL excels at representing functions containing high-frequency information. Specifically, the neural networks learned in each grade adeptly capture some low-frequency information, allowing their compositions with SNNs learned in the previous grades effectively representing the high-frequency features. Our experimental results underscore the efficacy of MGDL in addressing the spectral bias inherent in DNNs. By leveraging MGDL, we offer insights into overcoming spectral bias limitation of DNNs, thereby enhancing the performance and applicability of deep learning models in tasks requiring the representation of high-frequency information. This study confirms that the proposed method offers a promising solution to address the spectral bias of DNNs.

On Spectral Bias Reduction of Multi-scale Neural Networks for Regression Problems

Frequency-adaptive Multi-scale Deep Neural Networks

Multi-scale Deep Neural Networks for Solving High Dimensional PDEs

Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning

A deep neural network approach on solving the linear transport model under diffusive scaling

Spectrum-Informed Multistage Neural Networks: Multiscale Function Approximators of Machine Precision

Subspace decomposition based DNN algorithm for elliptic type multi-scale PDEs

Scaling Laws in Linear Regression: Compute, Parameters, and Data

Multi-scale Deep Neural Network (mscalednn) for Solving Poisson-Boltzmann Equation in Complex Domains

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

A Multi-Scale DNN Algorithm for Nonlinear Elliptic Equations with Multiple Scales

Mitigating spectral bias for the multiscale operator learning

On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks

Diffusion-based Semi-supervised Spectral Algorithm for Regression on Manifolds

ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connection

Multi-stage Neural Networks: Function Approximator of Machine Precision

Solving multiscale elliptic problems by sparse radial basis function neural networks

Multi-scale Feature Learning Dynamics: Insights for Double Descent

Spectral methods for Neural Integral Equations

A Spectral Theory of Neural Prediction and Alignment

The Multiscale Structure of Neural Network Loss Functions: The Effect on Optimization and Origin