Abstract:Effective activation functions introduce non-linear transformations, providing neural networks with stronger fitting capa-bilities, which help them better adapt to real data distributions. Huawei Noah's Lab believes that dynamic activation functions are more suitable than static activation functions for enhancing the non-linear capabilities of neural networks. Tsinghua University's related research also suggests using dynamically adjusted activation functions. Building on the ideas of using fine-tuned activation functions from Tsinghua University and Huawei Noah's Lab, we propose a series-based learnable ac-tivation function called LSLU (Learnable Series Linear Units). This method simplifies deep learning networks while im-proving accuracy. This method introduces learnable parameters {\theta} and {\omega} to control the activation function, adapting it to the current layer's training stage and improving the model's generalization. The principle is to increase non-linearity in each activation layer, boosting the network's overall non-linearity. We evaluate LSLU's performance on CIFAR10, CIFAR100, and specific task datasets (e.g., Silkworm), validating its effectiveness. The convergence behavior of the learnable parameters {\theta} and {\omega}, as well as their effects on generalization, are analyzed. Our empirical results show that LSLU enhances the general-ization ability of the original model in various tasks while speeding up training. In VanillaNet training, parameter {\theta} initially decreases, then increases before stabilizing, while {\omega} shows an opposite trend. Ultimately, LSLU achieves a 3.17% accuracy improvement on CIFAR100 for VanillaNet (Table 3). Codes are available at <a class="link-external link-https" href="https://github.com/vontran2021/Learnable-series-linear-units-LSLU" rel="external noopener nofollow">this https URL</a>.

Evaluating CNN with Oscillatory Activation Function

Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networks

ASU-CNN: An Efficient Deep Architecture for Image Classification and Feature Visualizations

Trainable Activation Function in Image Classification

Design of CNNs with adaptive activation function and application in fault detection

A Non-monotonic Smooth Activation Function

The Influence of the Activation Function in a Convolution Neural Network Model of Facial Expression Recognition

Amplifying Sine Unit: An Oscillatory Activation Function for Deep Neural Networks to Recover Nonlinear Oscillations Efficiently

Normalized Activation Function: Toward Better Convergence

PFLU and FPFLU: Two novel non-monotonic activation functions in convolutional neural networks

Oscillations in an artificial neural network convert competing inputs into a temporal code

Learning continuous piecewise non-linear activation functions for deep neural networks

HOSC: A Periodic Activation Function for Preserving Sharp Features in Implicit Neural Representations

Activation Functions: Dive into an optimal activation function

Activation function optimization method: Learnable series linear units (LSLUs)

Non-linear Convolution Filters for CNN-based Learning

An overview of the activation functions used in deep learning algorithms

EIS - Efficient and Trainable Activation Functions for Better Accuracy and Performance

Ensemble of Convolutional Neural Networks Trained with Different Activation Functions

Learning Combinations of Activation Functions

Improved Learning by Using a Modified Activation Function of a Convolutional Neural Network in Image Classification