Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks

Vladimír Kunc,Jiří Kléma

2024-02-14

Abstract:Neural networks have proven to be a highly effective tool for solving complex problems in many areas of life. Recently, their importance and practical usability have further been reinforced with the advent of deep learning. One of the important conditions for the success of neural networks is the choice of an appropriate activation function introducing non-linearity into the model. Many types of these functions have been proposed in the literature in the past, but there is no single comprehensive source containing their exhaustive overview. The absence of this overview, even in our experience, leads to redundancy and the unintentional rediscovery of already existing activation functions. To bridge this gap, our paper presents an extensive survey involving 400 activation functions, which is several times larger in scale than previous surveys. Our comprehensive compilation also references these surveys; however, its main goal is to provide the most comprehensive overview and systematization of previously published activation functions with links to their original sources. The secondary aim is to update the current understanding of this family of functions.

Machine Learning,Neural and Evolutionary Computing

What problem does this paper attempt to address?

The paper attempts to address the issue of the selection and systematization of activation functions in neural networks. Although many activation functions have been proposed, there is a lack of a comprehensive summary list, leading researchers to often inadvertently reinvent existing activation functions, resulting in a waste of research resources. This paper fills this gap by providing a comprehensive survey that includes 400 activation functions, aiming to avoid redundant work and promote rapid progress in the research of activation functions in the field of neural networks. Additionally, the paper aims to update the current understanding of these families of functions. In summary, the goal of the paper is to provide a comprehensive and systematic list of activation functions to help researchers better choose the appropriate activation function for their neural network tasks.

Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks

Activation Functions: Comparison of trends in Practice and Research for Deep Learning

How important are activation functions in regression and classification? A survey, performance comparison, and future directions

A survey on modern trainable activation functions

Activation Functions in Artificial Neural Networks: A Systematic Overview

An overview of the activation functions used in deep learning algorithms

Efficient Activation Function Optimization through Surrogate Modeling

Normalized Activation Function: Toward Better Convergence

Activation Functions: Dive into an optimal activation function

Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functions

A Method on Searching Better Activation Functions

Activation Ensembles for Deep Neural Networks

Ensemble of Convolutional Neural Networks Trained with Different Activation Functions

A novel activation function for multilayer feed-forward neural networks

A Review of Activation Function for Artificial Neural Network

A Survey of Deep Learning: From Activations to Transformers

Activations Through Extensions: A Framework To Boost Performance Of Neural Networks

Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks

On Functional Activations in Deep Neural Networks

Heuristic Search for Activation Functions of Neural Networks Based on Gaussian Processes.

Learning Combinations of Activation Functions