Abstract:In the architecture of deep learning models, inspired by biological neurons, activation functions (AFs) play a pivotal role. They significantly influence the performance of artificial neural networks. By modulating the non-linear properties essential for learning complex patterns, AFs are fundamental in both classification and regression tasks. This paper presents a comprehensive review of various types of AFs, including fixed-shape, parametric, adaptive, stochastic/probabilistic, non-standard, and ensemble/combining types. We begin with a systematic taxonomy and detailed classification frameworks that delineates the principal characteristics of AFs and organizes them based on their structural and functional distinctions. Our in-depth analysis covers primary groups such as sigmoid-based, ReLU-based, and ELU-based AFs, discussing their theoretical foundations, mathematical formulations, and specific benefits and limitations in different contexts. We also highlight key attributes of AFs such as output range, monotonicity, and smoothness. Furthermore, we explore miscellaneous AFs that do not conform to these categories but have shown unique advantages in specialized applications. Non-standard AFs are also explored, showcasing cutting-edge variations that challenge traditional paradigms and offer enhanced adaptability and model performance. We examine strategies for combining multiple AFs to leverage complementary properties. The paper concludes with a comparative evaluation of 12 state-of-the-art AFs, using rigorous statistical and experimental methodologies to assess their efficacy. This analysis not only aids practitioners in selecting and designing the most appropriate AFs for their specific deep learning tasks but also encourages continued innovation in AF development within the machine learning community.

E-Tanh: a novel activation function for image processing neural network models

ErfReLU: Adaptive Activation Function for Deep Neural Network

Enhancement of Neural Networks with an Alternative Activation Function Tanhlu

An Efficient Asymmetric Nonlinear Activation Function for Deep Neural Networks

A novel activation function for multilayer feed-forward neural networks

EIS - Efficient and Trainable Activation Functions for Better Accuracy and Performance

Activation function optimization scheme for image classification

TaLU: A Hybrid Activation Function Combining Tanh and Rectified Linear Unit to Enhance Neural Networks

Deep Learning Activation Functions: Fixed-Shape, Parametric, Adaptive, Stochastic, Miscellaneous, Non-Standard, Ensemble

A comparison of activation functions in artificial neural networks

Developing Novel T-Swish Activation Function in Deep Learning

Smish: A Novel Activation Function for Deep Learning Methods

Full-Analog Implementation of Activation Function Based on Phase-Change Memory for Artificial Neural Networks

A Method on Searching Better Activation Functions

An overview of the activation functions used in deep learning algorithms

Trainable Highly-expressive Activation Functions

Neural network with NewSigmoid activation function

S-Shape Activation Function with Adaptive Saturation Rate for Neural Networks

The Convolution Neural Network with Transformed Exponential Linear Unit Activation Function for Image Classification

Influence of Different Activation Functions on Deep Learning Models in Indoor Scene Images Classification