Abstract: Activation functions are essential for neural networks to introduce non-linearity. A great number of empirical experiments have validated various activation functions, yet theoretical research on activation functions are insufficient. In this work, we study the impact of activation functions on the variance of gradients and propose an approach to normalize activation functions to keep the variance of the gradient same for all layers so that the neural network can achieve better convergence. First, we complement the previous work on the analysis of the variance of gradients where the impact of activation functions are just considered in an idealized initial state which almost cannot be preserved during training and obtained a property that good activation functions should satisfy as possible. Second, we offer an approach to normalize activation functions and testify its effectiveness on prevalent activation functions empirically. And by observing experiments, we discover that the speed of convergence is roughly related to the property we derived in the former part. We run experiments of our normalized activation functions against common activation functions. And the result shows our approach consistently outperforms their unnormalized counterparts. For example, normalized Swish outperforms vanilla Swish by 1.2% on ResNet50 with CIFAR-100 in terms of top-1 accuracy. Our method improves the performance by simply replacing activation functions with their normalized ones in both fully-connected networks and residual networks.

Analysis on Cellular Neural Networks with Negative Slope Activation Function

The Existence and Global Attractivity of Almost Periodic Sequence Solution of Discrete-Time Neural Networks

Convergence Analysis of General Neural Networks under Almost Periodic Stimuli

A New Lmi-Based Approach For Stability Analysis Of Cellular Neural Networks With Time Delay

Sufficient condition of stability for cellular neural networks

Normalized Activation Function: Toward Better Convergence

Stability analysis on a neutral neural network model

Novel Exponential Stability and Periodicity of Cellular Neural Networks

Exponential Stability for Cellular Neural Networks: an LMI Approach

ANAct: Adaptive Normalization for Activation Functions

Trainable activation function with differentiable negative side and adaptable rectified point

On Global Exponential Stability Of Cellular Neural Networks With Lipschitz-Continuous Activation Function And Variable Delays

Multistability and complete convergence analysis on high-order neural networks with a class of nonsmooth activation functions

Bifurcation And Chaos Analysis For A Delayed Two-Neural Network With A Variation Slope Ratio In The Activation Function

Stability Analysis for Neural Networks with Inverse Lipschitzian Neuron Activations and Impulses

Equilibrium Points and Stability Analysis of a Class of Neural Networks

R-+(N)-Global Stability of A Cohen-Grossberg Neural Network System with Nonnegative Equilibria

Exponential Stability Analysis of Stochastic Delayed Cellular Neural Network

Stability Analysis on a Class of Nonlinear Continuous Neural Networks

Dynamical Analysis of Neural Networks of Subgradient System

Learn to Enhance the Negative Information in Convolutional Neural Network