Abstract:Activation functions play a key role in providing remarkable performance in deep neural networks, and the rectified linear unit (ReLU) is one of the most widely used activation functions. Various new activation functions and improvements on ReLU have been proposed, but each carry performance drawbacks. In this paper, we propose an improved activation function, which we name the natural-logarithm-rectified linear unit (NLReLU). This activation function uses the parametric natural logarithmic transform to improve ReLU and is simply defined as. NLReLU not only retains the sparse activation characteristic of ReLU, but it also alleviates the "dying ReLU" and vanishing gradient problems to some extent. It also reduces the bias shift effect and heteroscedasticity of neuron data distributions among network layers in order to accelerate the learning process. The proposed method was verified across ten convolutional neural networks with different depths for two essential datasets. Experiments illustrate that convolutional neural networks with NLReLU exhibit higher accuracy than those with ReLU, and that NLReLU is comparable to other well-known activation functions. NLReLU provides 0.16% and 2.04% higher classification accuracy on average compared to ReLU when used in shallow convolutional neural networks with the MNIST and CIFAR-10 datasets, respectively. The average accuracy of deep convolutional neural networks with NLReLU is 1.35% higher on average with the CIFAR-10 dataset.

SignReLU neural network and its approximation ability

Neural networks with ReLU powers need less depth

Improving Fault Tolerance for Reliable DNN Using Boundary-Aware Activation

Natural-Logarithm-Rectified Activation Function in Convolutional Neural Networks

Optimal function approximation with ReLU neural networks

ReLUs Are Sufficient for Learning Implicit Neural Representations

ReLU soothes the NTK condition number and accelerates optimization for wide neural networks

Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class.

UR4NNV: Neural Network Verification, Under-approximation Reachability Works!

Optimal rates of approximation by shallow ReLU$^k$ neural networks and applications to nonparametric regression

On the Expressive Power of Neural Networks

Competition-based Adaptive ReLU for Deep Neural Networks

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

MMReLU: A Simple and Smooth Activation Function with High Convergence Speed

Optimal Rates of Approximation by Shallow ReLU Neural Networks and Applications to Nonparametric Regression

ReLU neural network approximation to piecewise constant functions

Uniform Approximation with Quadratic Neural Networks

Deep Learning with S-shaped Rectified Linear Activation Units

LogReLU:A New Rectified Activation Unit Using Log Funcion

FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks

A Method to Improve the Computational Performance of Nonlinear All—optical Diffractive Deep Neural Network Model