Abstract:Activation functions are crucial in deep learning networks, given that the nonlinear ability of activation functions endows deep neural networks with real artificial intelligence. Nonlinear nonmonotonic activation functions, such as rectified linear units, Tan hyperbolic (tanh), Sigmoid, Swish, Mish, and Logish, perform well in deep learning models; however, only a few of them are widely used in mostly all applications due to their existing inconsistencies. Inspired by the MB-C-BSIF method, this study proposes Smish, a novel nonlinear activation function, expressed as f(x)=x·tanh[ln(1+sigmoid(x))], which could overcome other activation functions with good properties. Logarithmic operations are first used to reduce the range of sigmoid(x). The value is then calculated using the tanh operator. Inputs are ultimately used to multiply the previous value, thus exhibiting negative output regularization. Experiments show that Smish tends to operate more efficiently than Logish, Mish, and other activation functions on EfficientNet models with open datasets. Moreover, we evaluated the performance of Smish in various deep learning models and the parameters of its function f(x)=αx·tanh[ln(1+sigmoid(βx))], and where α = 1 and β = 1, Smish was found to exhibit the highest accuracy. The experimental results show that with Smish, the EfficientNetB3 network exhibits a Top-1 accuracy of 84.1% on the CIFAR-10 dataset; the EfficientNetB5 network has a Top-1 accuracy of 99.89% on the MNIST dataset; and the EfficientnetB7 network has a Top-1 accuracy of 91.14% on the SVHN dataset. These values are superior to those obtained using other state-of-the-art activation functions, which shows that Smish is more suitable for complex deep learning models.

MMReLU: A Simple and Smooth Activation Function with High Convergence Speed

Normalized Activation Function: Toward Better Convergence

Improving Fault Tolerance for Reliable DNN Using Boundary-Aware Activation

Mish: A Self Regularized Non-Monotonic Neural Activation Function

SMU: smooth activation function for deep networks using smoothing maximum technique

Competition-based Adaptive ReLU for Deep Neural Networks

Saturated Non-Monotonic Activation Functions

Deep Learning with S-shaped Rectified Linear Activation Units

A Non-monotonic Smooth Activation Function

Generalized Activation Via Multivariate Projection

FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks

Moderate Adaptive Linear Units (MoLU)

Improved Convolutional Neural Network Based on Fast Exponentially Linear Unit Activation Function.

Symmetric-threshold ReLU for Fast and Nearly Lossless ANN-SNN Conversion

Optimizing cnn-Bigru performance: Mish activation and comparative analysis with Relu

Smooth activations and reproducibility in deep networks

Smish: A Novel Activation Function for Deep Learning Methods

DSReLU: A Novel Dynamic Slope Function for Superior Model Training

Convergence of Deep Neural Networks with General Activation Functions and Pooling

Effects of the Nonlinearity in Activation Functions on the Performance of Deep Learning Models

An Efficient Asymmetric Nonlinear Activation Function for Deep Neural Networks