Abstract:In the architecture of deep learning models, inspired by biological neurons, activation functions (AFs) play a pivotal role. They significantly influence the performance of artificial neural networks. By modulating the non-linear properties essential for learning complex patterns, AFs are fundamental in both classification and regression tasks. This paper presents a comprehensive review of various types of AFs, including fixed-shape, parametric, adaptive, stochastic/probabilistic, non-standard, and ensemble/combining types. We begin with a systematic taxonomy and detailed classification frameworks that delineates the principal characteristics of AFs and organizes them based on their structural and functional distinctions. Our in-depth analysis covers primary groups such as sigmoid-based, ReLU-based, and ELU-based AFs, discussing their theoretical foundations, mathematical formulations, and specific benefits and limitations in different contexts. We also highlight key attributes of AFs such as output range, monotonicity, and smoothness. Furthermore, we explore miscellaneous AFs that do not conform to these categories but have shown unique advantages in specialized applications. Non-standard AFs are also explored, showcasing cutting-edge variations that challenge traditional paradigms and offer enhanced adaptability and model performance. We examine strategies for combining multiple AFs to leverage complementary properties. The paper concludes with a comparative evaluation of 12 state-of-the-art AFs, using rigorous statistical and experimental methodologies to assess their efficacy. This analysis not only aids practitioners in selecting and designing the most appropriate AFs for their specific deep learning tasks but also encourages continued innovation in AF development within the machine learning community.

S-Shape Activation Function with Adaptive Saturation Rate for Neural Networks

Stochastic Adaptive Activation Function

A novel activation function for multilayer feed-forward neural networks

An Efficient Asymmetric Nonlinear Activation Function for Deep Neural Networks

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Normalized Activation Function: Toward Better Convergence

Adaptive Parametric Activation

Activation Adaptation in Neural Networks

Neural network with NewSigmoid activation function

Learning Activation Functions for Sparse Neural Networks

Activate or Not: Learning Customized Activation

PATS: A New Neural Network Activation Function with Parameter.

Deep Learning Activation Functions: Fixed-Shape, Parametric, Adaptive, Stochastic, Miscellaneous, Non-Standard, Ensemble

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

Locally adaptive activation functions with slope recovery term for deep and physics-informed neural networks

Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks

Data-specific activation function learning for constructive neural networks

A New Kind of Neuron Model with A Tunable Activation Function and Its Applications

APALU: A Trainable, Adaptive Activation Function for Deep Learning Networks

Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functions

A New Artificial Neuron Proposal with Trainable Simultaneous Local and Global Activation Function