Abstract:Traditional activation functions such as hyperbolic tangent and logistic sigmoid have seen frequent use historically in artificial neural networks. However, nowadays, in practice, they have fallen out of favor, undoubtedly due to the gap in performance observed in recognition and classification tasks when compared to their well-known counterparts such as rectified linear or maxout. In this paper, we introduce a simple, new type of activation function for multilayer feed-forward architectures. Unlike other approaches where new activation functions have been designed by discarding many of the mainstays of traditional activation function design, our proposed function relies on them and therefore shares most of the properties found in traditional activation functions. Nevertheless, our activation function differs from traditional activation functions on two major points: its asymptote and global extremum. Defining a function which enjoys the property of having a global maximum and minimum, turned out to be critical during our design-process since we believe it is one of the main reasons behind the gap observed in performance between traditional activation functions and their recently introduced counterparts. We evaluate the effectiveness of the proposed activation function on four commonly used datasets, namely, MNIST, CIFAR-10, CIFAR-100, and the Pang and Lee’s movie review. Experimental results demonstrate that the proposed function can effectively be applied across various datasets where our accuracy, given the same network topology, is competitive with the state-of-the-art. In particular, the proposed activation function outperforms the state-of-the-art methods on the MNIST dataset.

An Efficient Asymmetric Nonlinear Activation Function for Deep Neural Networks

ErfReLU: Adaptive Activation Function for Deep Neural Network

A novel activation function for multilayer feed-forward neural networks

Normalized Activation Function: Toward Better Convergence

E-Tanh: a novel activation function for image processing neural network models

Symmetric-threshold ReLU for Fast and Nearly Lossless ANN-SNN Conversion

Nonlinearity Enhanced Adaptive Activation Function

Deep Networks with Non-Static Activation Function

EIS - Efficient and Trainable Activation Functions for Better Accuracy and Performance

A Method on Searching Better Activation Functions

APALU: A Trainable, Adaptive Activation Function for Deep Learning Networks

Activate or Not: Learning Customized Activation

Performance Analysis of Nonlinear Activation Function in Convolution Neural Network for Image Classification

S-Shape Activation Function with Adaptive Saturation Rate for Neural Networks

ASU-CNN: An Efficient Deep Architecture for Image Classification and Feature Visualizations

Competition-based Adaptive ReLU for Deep Neural Networks

ANAct: Adaptive Normalization for Activation Functions

ENN: A Neural Network with DCT Adaptive Activation Functions

Efficient Nonlinear Activation Function in Optical Neural Network