Abstract:Motivated by the growing theoretical understanding of neural networks that employ the Rectified Linear Unit (ReLU) as their activation function, we revisit the use of ReLU activation functions for learning implicit neural representations (INRs). Inspired by second order B-spline wavelets, we incorporate a set of simple constraints to the ReLU neurons in each layer of a deep neural network (DNN) to remedy the spectral bias. This in turn enables its use for various INR tasks. Empirically, we demonstrate that, contrary to popular belief, one can learn state-of-the-art INRs based on a DNN composed of only ReLU neurons. Next, by leveraging recent theoretical works which characterize the kinds of functions ReLU neural networks learn, we provide a way to quantify the regularity of the learned function. This offers a principled approach to selecting the hyperparameters in INR architectures. We substantiate our claims through experiments in signal representation, super resolution, and computed tomography, demonstrating the versatility and effectiveness of our method. The code for all experiments can be found at <a class="link-external link-https" href="https://github.com/joeshenouda/relu-inrs" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The paper primarily aims to address the following two core issues: 1. **Overcoming the limitations of the ReLU activation function in Implicit Neural Representations (INRs) tasks**: The traditional ReLU activation function, due to its inherent spectral bias, performs poorly in approximating high-frequency signals. This limitation restricts its application in INR tasks that require efficient capture of high-frequency components in images. To overcome this limitation, the authors propose a method that constructs a network structure capable of effectively mitigating spectral bias by imposing a set of simple constraints on ReLU neurons within deep neural networks (DNNs). This improved network is referred to as the BW-ReLU neural network. 2. **Quantifying and understanding the generalization ability of INRs**: By leveraging recent theoretical advancements to characterize the types of functions learned by ReLU neural networks, the paper further proposes methods to quantify the regularity of these functions, particularly for the proposed BW-ReLU neural network. This regularity can be measured using the variation norm, which not only provides an understanding of the smoothness of the learned functions but also reveals some principles behind commonly used empirical rules in training INRs, such as the impact of introducing a scale parameter \(c\). Experiments show that BW-ReLU neural networks with lower variation norms tend to generalize better to unseen data. In summary, this research aims to demonstrate how a novel approach—imposing specific constraints on ReLU neurons to construct BW-ReLU neural networks—can overcome the limitations of traditional ReLU DNNs in INR tasks, and provides a new perspective for evaluating and optimizing the generalization performance of these networks.

ReLUs Are Sufficient for Learning Implicit Neural Representations

Single-Layer Learnable Activation for Implicit Neural Representation (SL$^{2}$A-INR)

Neural networks with ReLU powers need less depth

Where Do We Stand with Implicit Neural Representations? A Technical and Performance Survey

Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics

Deep Learning with S-shaped Rectified Linear Activation Units

Deep Representation with ReLU Neural Networks

Deep Neural Networks with Elastic Rectified Linear Units for Object Recognition

Nonparametric regression using deep neural networks with ReLU activation function

Taming the ReLU with Parallel Dither in a Deep Neural Network

Revisiting Implicit Neural Representations in Low-Level Vision

Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training

Towards a Sampling Theory for Implicit Neural Representations

Implicit Neural Representations and the Algebra of Complex Wavelets

SignReLU neural network and its approximation ability

A Structured Dictionary Perspective on Implicit Neural Representations

ReLU soothes the NTK condition number and accelerates optimization for wide neural networks

Image Denoising with Rectified Linear Units.

Multi-Bias Non-linear Activation in Deep Neural Networks

H-SIREN: Improving implicit neural representations with hyperbolic periodic functions

Natural-Logarithm-Rectified Activation Function in Convolutional Neural Networks