Abstract:Purpose The purpose of this study is to explore the potential of trainable activation functions to enhance the performance of deep neural networks, specifically ResNet architectures, in the task of image classification. By introducing activation functions that adapt during training, the authors aim to determine whether such flexibility can lead to improved learning outcomes and generalization capabilities compared to static activation functions like ReLU. This research seeks to provide insights into how dynamic nonlinearities might influence deep learning models' efficiency and accuracy in handling complex image data sets. Design/methodology/approach This research integrates three novel trainable activation functions – CosLU, DELU and ReLUN – into various ResNet-n architectures, where "n" denotes the number of convolutional layers. Using CIFAR-10 and CIFAR-100 data sets, the authors conducted a comparative study to assess the impact of these functions on image classification accuracy. The approach included modifying the traditional ResNet models by replacing their static activation functions with the trainable variants, allowing for dynamic adaptation during training. The performance was evaluated based on accuracy metrics and loss profiles across different network depths. Findings The findings indicate that trainable activation functions, particularly CosLU, can significantly enhance the performance of deep learning models, outperforming the traditional ReLU in deeper network configurations on the CIFAR-10 data set. CosLU showed the highest improvement in accuracy, whereas DELU and ReLUN offered varying levels of performance enhancements. These functions also demonstrated potential in reducing overfitting and improving model generalization across more complex data sets like CIFAR-100, suggesting that the adaptability of activation functions plays a crucial role in the training dynamics of deep neural networks. Originality/value This study contributes to the field of deep learning by introducing and evaluating the impact of three novel trainable activation functions within widely used ResNet architectures. Unlike previous works that primarily focused on static activation functions, this research demonstrates that incorporating trainable nonlinearities can lead to significant improvements in model performance and adaptability. The introduction of CosLU, DELU and ReLUN provides a new pathway for enhancing the flexibility and efficiency of neural networks, potentially setting a new standard for future deep learning applications in image classification and beyond.

Improved Learning by Using a Modified Activation Function of a Convolutional Neural Network in Image Classification

Improved learning by using a modified activation function of a Convolutional Neural Network in multi-spectral image classification

A Hyperspectral Image Classification Method Based on the Nonlocal Attention Mechanism of a Multiscale Convolutional Neural Network.

A lightweight and stochastic depth residual attention network for remote sensing scene classification

Multi-stack hybrid CNN with non-monotonic activation functions for hyperspectral satellite image classification

TaLU: A Hybrid Activation Function Combining Tanh and Rectified Linear Unit to Enhance Neural Networks

Web-aided data set expansion in deep learning: evaluating trainable activation functions in ResNet for improved image classification

FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks

Competition-based Adaptive ReLU for Deep Neural Networks

Comparison of Land Use Classification Based on Convolutional Neural Network

STRATEGIC OPTIMIZATION OF CONVOLUTIONAL NEURAL NETWORKS FOR HYPERSPECTRAL LAND COVER CLASSIFICATION

Improved Convolutional Neural Network Based on Fast Exponentially Linear Unit Activation Function.

Minimum Noise Fraction and Long Short-Term Memory Model for Hyperspectral Imaging

Spatially regularized Leaky ReLU in dual space for CNN based image segmentation

The Influence of the Activation Function in a Convolution Neural Network Model of Facial Expression Recognition

Learn to Enhance the Negative Information in Convolutional Neural Network

An optimal model using hybrid LCNN- GRU for efficient hyperspectral image classification

Land-cover classification with hyperspectral remote sensing image using CNN and spectral band selection

Deep Transfer Learning for Land Use and Land Cover Classification: A Comparative Study

Learning continuous piecewise non-linear activation functions for deep neural networks

Optimized convolutional neural network for land cover classification via improved lion algorithm