Abstract:Activation function has a significant impact on the dynamics, convergence, and performance of deep neural networks. The search for a consistent and high-performing activation function has always been a pursuit during deep learning model development. Existing state-of-the-art activation functions are manually designed with human expertise except for Swish. Swish was developed using a reinforcement learning-based search strategy. In this study, we propose an evolutionary approach for optimizing activation functions specifically for image classification tasks, aiming to discover functions that outperform current state-of-the-art options. Through this optimization framework, we obtain a series of high-performing activation functions denoted as Exponential Error Linear Unit (EELU). The developed activation functions are evaluated for image classification tasks from two perspectives: (1) five state-of-the-art neural network architectures, such as ResNet50, AlexNet, VGG16, MobileNet, and Compact Convolutional Transformer which cover computationally heavy to light neural networks, and (2) eight standard datasets, including CIFAR10, Imagenette, MNIST, Fashion MNIST, Beans, Colorectal Histology, CottonWeedID15, and TinyImageNet which cover from typical machine vision benchmark, agricultural image applications to medical image applications. Finally, we statistically investigate the generalization of the resultant activation functions developed through the optimization scheme. With a Friedman test, we conclude that the optimization scheme is able to generate activation functions that outperform the existing standard ones in 92.8% cases among 28 different cases studied, and −x⋅erf(e−x) is found to be the best activation function for image classification generated by the optimization scheme.

Swish: a Self-Gated Activation Function

SwishReLU: A Unified Approach to Activation Functions for Enhanced Deep Neural Networks Performance

Developing Novel T-Swish Activation Function in Deep Learning

Smish: A Novel Activation Function for Deep Learning Methods

Swim: A General-Purpose, High-Performing, and Efficient Activation Function for Locomotion Control Tasks

Evaluating Model Performance with Hard-Swish Activation Function Adjustments

A Non-monotonic Smooth Activation Function

Rethinking the Activation Function in Lightweight Network.

Normalized Activation Function: Toward Better Convergence

Activate or Not: Learning Customized Activation

Mish: A Self Regularized Non-Monotonic Neural Activation Function

SMU: smooth activation function for deep networks using smoothing maximum technique

Logish: A New Nonlinear Nonmonotonic Activation Function for Convolutional Neural Network

Swish-T : Enhancing Swish Activation with Tanh Bias for Improved Neural Network Performance

Activation function optimization scheme for image classification

SERF: Towards better training of deep neural networks using log-Softplus ERror activation Function

EIS - Efficient and Trainable Activation Functions for Better Accuracy and Performance

Stochastic Adaptive Activation Function

Activation Functions: Dive into an optimal activation function

Zorro: A Flexible and Differentiable Parametric Family of Activation Functions That Extends ReLU and GELU

MMReLU: A Simple and Smooth Activation Function with High Convergence Speed