Abstract:Effective activation functions introduce non-linear transformations, providing neural networks with stronger fitting capa-bilities, which help them better adapt to real data distributions. Huawei Noah's Lab believes that dynamic activation functions are more suitable than static activation functions for enhancing the non-linear capabilities of neural networks. Tsinghua University's related research also suggests using dynamically adjusted activation functions. Building on the ideas of using fine-tuned activation functions from Tsinghua University and Huawei Noah's Lab, we propose a series-based learnable ac-tivation function called LSLU (Learnable Series Linear Units). This method simplifies deep learning networks while im-proving accuracy. This method introduces learnable parameters {\theta} and {\omega} to control the activation function, adapting it to the current layer's training stage and improving the model's generalization. The principle is to increase non-linearity in each activation layer, boosting the network's overall non-linearity. We evaluate LSLU's performance on CIFAR10, CIFAR100, and specific task datasets (e.g., Silkworm), validating its effectiveness. The convergence behavior of the learnable parameters {\theta} and {\omega}, as well as their effects on generalization, are analyzed. Our empirical results show that LSLU enhances the general-ization ability of the original model in various tasks while speeding up training. In VanillaNet training, parameter {\theta} initially decreases, then increases before stabilizing, while {\omega} shows an opposite trend. Ultimately, LSLU achieves a 3.17% accuracy improvement on CIFAR100 for VanillaNet (Table 3). Codes are available at <a class="link-external link-https" href="https://github.com/vontran2021/Learnable-series-linear-units-LSLU" rel="external noopener nofollow">this https URL</a>.

Non-uniform Piecewise Linear Activation Functions in Deep Neural Networks

Learning continuous piecewise non-linear activation functions for deep neural networks

A Flexible and High-Precision Activation Function Unit Based on Equi-Error Partitioning Algorithm

PFLU and FPFLU: Two novel non-monotonic activation functions in convolutional neural networks

Parametric Deformable Exponential Linear Units for Deep Neural Networks.

PWL- Explorer: A Reconfigurable Architecture for Nonlinear Activation Function with Automatic DSE

Parametric Variational Linear Units (PVLUs) in Deep Convolutional Networks

Piecewise Linear Neural Networks and Deep Learning

Activation function optimization method: Learnable series linear units (LSLUs)

On the Importance of Normalisation Layers in Deep Learning with Piecewise Linear Activation Units

APALU: A Trainable, Adaptive Activation Function for Deep Learning Networks

Moderate Adaptive Linear Units (MoLU)

Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks

Training neural networks for solving 1-D optimal piecewise linear approximation

PATS: A New Neural Network Activation Function with Parameter.

Leveraging Product as an Activation Function in Deep Networks

Normalized Activation Function: Toward Better Convergence

On the Number of Linear Regions of Convolutional Neural Networks with Piecewise Linear Activations.

Deep Global-Connected Net With The Generalized Multi-Piecewise ReLU Activation in Deep Learning

Improving Deep Neural Network with Multiple Parametric Exponential Linear Units

Nearly-tight bounds on linear regions of piecewise linear neural networks