Abstract:In the architecture of deep learning models, inspired by biological neurons, activation functions (AFs) play a pivotal role. They significantly influence the performance of artificial neural networks. By modulating the non-linear properties essential for learning complex patterns, AFs are fundamental in both classification and regression tasks. This paper presents a comprehensive review of various types of AFs, including fixed-shape, parametric, adaptive, stochastic/probabilistic, non-standard, and ensemble/combining types. We begin with a systematic taxonomy and detailed classification frameworks that delineates the principal characteristics of AFs and organizes them based on their structural and functional distinctions. Our in-depth analysis covers primary groups such as sigmoid-based, ReLU-based, and ELU-based AFs, discussing their theoretical foundations, mathematical formulations, and specific benefits and limitations in different contexts. We also highlight key attributes of AFs such as output range, monotonicity, and smoothness. Furthermore, we explore miscellaneous AFs that do not conform to these categories but have shown unique advantages in specialized applications. Non-standard AFs are also explored, showcasing cutting-edge variations that challenge traditional paradigms and offer enhanced adaptability and model performance. We examine strategies for combining multiple AFs to leverage complementary properties. The paper concludes with a comparative evaluation of 12 state-of-the-art AFs, using rigorous statistical and experimental methodologies to assess their efficacy. This analysis not only aids practitioners in selecting and designing the most appropriate AFs for their specific deep learning tasks but also encourages continued innovation in AF development within the machine learning community.

Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functions

Trainable Highly-expressive Activation Functions

Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks

Normalized Activation Function: Toward Better Convergence

APALU: A Trainable, Adaptive Activation Function for Deep Learning Networks

Adaptive Parametric Activation

Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks

Activation Functions: Comparison of trends in Practice and Research for Deep Learning

ErfReLU: Adaptive Activation Function for Deep Neural Network

A novel activation function for multilayer feed-forward neural networks

A New Kind of Neuron Model with A Tunable Activation Function and Its Applications

A survey on modern trainable activation functions

A Method on Searching Better Activation Functions

Activate or Not: Learning Customized Activation

An Efficient Asymmetric Nonlinear Activation Function for Deep Neural Networks

Your Network May Need to Be Rewritten: Network Adversarial Based on High-Dimensional Function Graph Decomposition

Activation Ensembles for Deep Neural Networks

An overview of the activation functions used in deep learning algorithms

Activation Functions in Artificial Neural Networks: A Systematic Overview

Deep Learning Activation Functions: Fixed-Shape, Parametric, Adaptive, Stochastic, Miscellaneous, Non-Standard, Ensemble