Abstract:Deep neural networks (DNNs) suffer from the spectral bias, wherein DNNs typically exhibit a tendency to prioritize the learning of lower-frequency components of a function, struggling to capture its high-frequency features. This paper is to address this issue. Notice that a function having only low frequency components may be well-represented by a shallow neural network (SNN), a network having only a few layers. By observing that composition of low frequency functions can effectively approximate a high-frequency function, we propose to learn a function containing high-frequency components by composing several SNNs, each of which learns certain low-frequency information from the given data. We implement the proposed idea by exploiting the multi-grade deep learning (MGDL) model, a recently introduced model that trains a DNN incrementally, grade by grade, a current grade learning from the residue of the previous grade only an SNN composed with the SNNs trained in the preceding grades as features. We apply MGDL to synthetic, manifold, colored images, and MNIST datasets, all characterized by presence of high-frequency features. Our study reveals that MGDL excels at representing functions containing high-frequency information. Specifically, the neural networks learned in each grade adeptly capture some low-frequency information, allowing their compositions with SNNs learned in the previous grades effectively representing the high-frequency features. Our experimental results underscore the efficacy of MGDL in addressing the spectral bias inherent in DNNs. By leveraging MGDL, we offer insights into overcoming spectral bias limitation of DNNs, thereby enhancing the performance and applicability of deep learning models in tasks requiring the representation of high-frequency information. This study confirms that the proposed method offers a promising solution to address the spectral bias of DNNs.

A Uniform Framework of Yau-Yau Algorithm Based on Deep Learning with the Capability of Overcoming the Curse of Dimensionality

Deep Filtering With Adaptive Learning Rates

DDPG-Driven Deep-Unfolding with Adaptive Depth for Channel Estimation with Sparse Bayesian Learning

Deep Nonlinear Sufficient Dimension Reduction

On the Convergence Analysis of Yau-Yau Nonlinear Filtering Algorithm: from a Probabilistic Perspective

Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class.

An Efficient Algorithm of Yau-Yau Method for Solving Nonlinear Filtering Problems.

Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing

An Efficient Numerical Method for Solving High-Dimensional Nonlinear Filtering Problems.

Designing Universally-Approximating Deep Neural Networks: A First-Order Optimization Approach

Near-optimal learning of Banach-valued, high-dimensional functions via deep neural networks

A Simple And Effective Filtering Scheme For Improving Neural Fields

A Multi-parameter Updating Fourier Online Gradient Descent Algorithm for Large-scale Nonlinear Classification

An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network

A Scalable Deep Learning Approach for Solving High-Dimensional Dynamic Optimal Transport.

Features are fate: a theory of transfer learning in high-dimensional regression

A proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients

Convergence Framework of Deep Learning-based Hybrid Iterative Methods and the Application to Designing a Fourier Neural Solver for Parametric PDEs

Deep Transfer Learning: Model Framework and Error Analysis

Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks

Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning