Abstract:Effective activation functions introduce non-linear transformations, providing neural networks with stronger fitting capa-bilities, which help them better adapt to real data distributions. Huawei Noah's Lab believes that dynamic activation functions are more suitable than static activation functions for enhancing the non-linear capabilities of neural networks. Tsinghua University's related research also suggests using dynamically adjusted activation functions. Building on the ideas of using fine-tuned activation functions from Tsinghua University and Huawei Noah's Lab, we propose a series-based learnable ac-tivation function called LSLU (Learnable Series Linear Units). This method simplifies deep learning networks while im-proving accuracy. This method introduces learnable parameters {\theta} and {\omega} to control the activation function, adapting it to the current layer's training stage and improving the model's generalization. The principle is to increase non-linearity in each activation layer, boosting the network's overall non-linearity. We evaluate LSLU's performance on CIFAR10, CIFAR100, and specific task datasets (e.g., Silkworm), validating its effectiveness. The convergence behavior of the learnable parameters {\theta} and {\omega}, as well as their effects on generalization, are analyzed. Our empirical results show that LSLU enhances the general-ization ability of the original model in various tasks while speeding up training. In VanillaNet training, parameter {\theta} initially decreases, then increases before stabilizing, while {\omega} shows an opposite trend. Ultimately, LSLU achieves a 3.17% accuracy improvement on CIFAR100 for VanillaNet (Table 3). Codes are available at <a class="link-external link-https" href="https://github.com/vontran2021/Learnable-series-linear-units-LSLU" rel="external noopener nofollow">this https URL</a>.

Logish: A New Nonlinear Nonmonotonic Activation Function for Convolutional Neural Network

Smish: A Novel Activation Function for Deep Learning Methods

Mish: A Self Regularized Non-Monotonic Neural Activation Function

A Non-monotonic Smooth Activation Function

Normalized Activation Function: Toward Better Convergence

Saturated Non-Monotonic Activation Functions

Nish: A Novel Negative Stimulated Hybrid Activation Function

Developing Novel T-Swish Activation Function in Deep Learning

Nonlinearity Enhanced Adaptive Activation Function

A Significantly Better Class of Activation Functions Than ReLU Like Activation Functions

Activation function optimization method: Learnable series linear units (LSLUs)

Activation function optimization scheme for image classification

An Efficient Asymmetric Nonlinear Activation Function for Deep Neural Networks

The Influence of the Activation Function in a Convolution Neural Network Model of Facial Expression Recognition

Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networks

SwishReLU: A Unified Approach to Activation Functions for Enhanced Deep Neural Networks Performance

EIS - Efficient and Trainable Activation Functions for Better Accuracy and Performance

Effects of the Nonlinearity in Activation Functions on the Performance of Deep Learning Models

Improved Convolutional Neural Network Based on Fast Exponentially Linear Unit Activation Function.

Logical Activation Functions: Logit-space equivalents of Probabilistic Boolean Operators

Learning continuous piecewise non-linear activation functions for deep neural networks