Parametric RSigELU: a new trainable activation function for deep learning

Serhat Kiliçarslan,Mete Celik

DOI: https://doi.org/10.1007/s00521-024-09538-9

2024-02-28

Neural Computing and Applications

Abstract:Abstract Activation functions are used to extract meaningful relationships from real-world problems with the help of deep learning models. Thus, the development of activation functions which affect deep learning models’ performances is of great interest to researchers. In the literature, mostly, nonlinear activation functions are preferred since linear activation functions limit the learning performances of the deep learning models. Non-linear activation functions can be classified as fixed-parameter and trainable activation functions based on whether the activation function parameter is fixed (i.e., user-given) or modified during the training process of deep learning models. The parameters of the fixed-parameter activation functions should be specified before the deep learning model training process. However, it takes too much time to determine appropriate function parameter values and can cause the slow convergence of the deep learning model. In contrast, trainable activation functions whose parameters are updated in each iteration of deep learning models training process achieve faster and better convergence by obtaining the most suitable parameter values for the datasets and deep learning architectures. This study proposes parametric RSigELU (P+RSigELU) trainable activation functions, such as P+RSigELU Single (P+RSigELUS) and P+RSigELU Double (P+RSigELUD), to improve the performance of fixed-parameter activation function of RSigELU. The performances of the proposed trainable activation functions were evaluated on the benchmark datasets of MNIST, CIFAR-10, and CIFAR-100 datasets. Results show that the proposed activation functions outperforms PReLU, PELU, ALISA, P+FELU, PSigmoid, and GELU activation functions found in the literature. To access the codes of the activation function; https://github.com/serhatklc/P-RsigELU-Activation-Function .

computer science, artificial intelligence

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This paper aims to address the issue of selecting activation functions in deep learning models. Specifically, the paper proposes a new trainable activation function—Parameterized RSigELU (P-RSigELU), including P-RSigELU Single (P-RSigELUS) and P-RSigELU Double (P-RSigELUD), to improve the performance of the existing fixed-parameter activation function RSigELU.

### Main Contributions of the Paper

1. Proposed the Parameterized RSigELU (P-RSigELU) trainable activation function, such as P-RSigELUS and P-RSigELUD.
2. Evaluated the performance of the proposed P-RSigELU activation function on the MNIST, CIFAR-10, and CIFAR-100 benchmark datasets using convolutional neural network (CNN) models.
3. Experimental results show that the proposed trainable activation function outperforms existing activation functions such as PReLU, PELU, ALISA, P-FELU, PSigmoid, and GELU in terms of performance.
4. The proposed P-RSigELU activation function works effectively in both positive and negative regions and addresses the issues of gradient vanishing and negative regions.
5. The P-RSigELU activation function adapts well to the model and has a low risk of overfitting.