Abstract:Effective activation functions introduce non-linear transformations, providing neural networks with stronger fitting capa-bilities, which help them better adapt to real data distributions. Huawei Noah's Lab believes that dynamic activation functions are more suitable than static activation functions for enhancing the non-linear capabilities of neural networks. Tsinghua University's related research also suggests using dynamically adjusted activation functions. Building on the ideas of using fine-tuned activation functions from Tsinghua University and Huawei Noah's Lab, we propose a series-based learnable ac-tivation function called LSLU (Learnable Series Linear Units). This method simplifies deep learning networks while im-proving accuracy. This method introduces learnable parameters {\theta} and {\omega} to control the activation function, adapting it to the current layer's training stage and improving the model's generalization. The principle is to increase non-linearity in each activation layer, boosting the network's overall non-linearity. We evaluate LSLU's performance on CIFAR10, CIFAR100, and specific task datasets (e.g., Silkworm), validating its effectiveness. The convergence behavior of the learnable parameters {\theta} and {\omega}, as well as their effects on generalization, are analyzed. Our empirical results show that LSLU enhances the general-ization ability of the original model in various tasks while speeding up training. In VanillaNet training, parameter {\theta} initially decreases, then increases before stabilizing, while {\omega} shows an opposite trend. Ultimately, LSLU achieves a 3.17% accuracy improvement on CIFAR100 for VanillaNet (Table 3). Codes are available at <a class="link-external link-https" href="https://github.com/vontran2021/Learnable-series-linear-units-LSLU" rel="external noopener nofollow">this https URL</a>.

A Flexible and High-Precision Activation Function Unit Based on Equi-Error Partitioning Algorithm

Improving Fault Tolerance for Reliable DNN Using Boundary-Aware Activation

Activation function optimization method: Learnable series linear units (LSLUs)

A High-Precision Flexible Symmetry-Aware Architecture for Element-Wise Activation Functions

APALU: A Trainable, Adaptive Activation Function for Deep Learning Networks

ReAFM: A Reconfigurable Nonlinear Activation Function Module for Neural Networks

Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks

A Method on Searching Better Activation Functions

PWL- Explorer: A Reconfigurable Architecture for Nonlinear Activation Function with Automatic DSE

Universal Activation Function For Machine Learning

An Efficient Asymmetric Nonlinear Activation Function for Deep Neural Networks

Regularized Flexible Activation Function Combinations for Deep Neural Networks

ErfReLU: Adaptive Activation Function for Deep Neural Network

Improved Convolutional Neural Network Based on Fast Exponentially Linear Unit Activation Function.

CombU: A Combined Unit Activation for Fitting Mathematical Expressions with Neural Networks

Reconfigurable Multifunction Computing Unit Using an Universal Piecewise Linear Method

Don't Fear Peculiar Activation Functions: EUAF and Beyond

Parametric Deformable Exponential Linear Units for Deep Neural Networks.

EIS - Efficient and Trainable Activation Functions for Better Accuracy and Performance

Normalized Activation Function: Toward Better Convergence