Abstract:Effective activation functions introduce non-linear transformations, providing neural networks with stronger fitting capa-bilities, which help them better adapt to real data distributions. Huawei Noah's Lab believes that dynamic activation functions are more suitable than static activation functions for enhancing the non-linear capabilities of neural networks. Tsinghua University's related research also suggests using dynamically adjusted activation functions. Building on the ideas of using fine-tuned activation functions from Tsinghua University and Huawei Noah's Lab, we propose a series-based learnable ac-tivation function called LSLU (Learnable Series Linear Units). This method simplifies deep learning networks while im-proving accuracy. This method introduces learnable parameters {\theta} and {\omega} to control the activation function, adapting it to the current layer's training stage and improving the model's generalization. The principle is to increase non-linearity in each activation layer, boosting the network's overall non-linearity. We evaluate LSLU's performance on CIFAR10, CIFAR100, and specific task datasets (e.g., Silkworm), validating its effectiveness. The convergence behavior of the learnable parameters {\theta} and {\omega}, as well as their effects on generalization, are analyzed. Our empirical results show that LSLU enhances the general-ization ability of the original model in various tasks while speeding up training. In VanillaNet training, parameter {\theta} initially decreases, then increases before stabilizing, while {\omega} shows an opposite trend. Ultimately, LSLU achieves a 3.17% accuracy improvement on CIFAR100 for VanillaNet (Table 3). Codes are available at <a class="link-external link-https" href="https://github.com/vontran2021/Learnable-series-linear-units-LSLU" rel="external noopener nofollow">this https URL</a>.

Data-specific activation function learning for constructive neural networks

Data-Driven Learning of Feedforward Neural Networks with Different Activation Functions

Activation Ensembles for Deep Neural Networks

A Method on Searching Better Activation Functions

Activate or Not: Learning Customized Activation

S-Shape Activation Function with Adaptive Saturation Rate for Neural Networks

A New Constructive Algorithm for Architectural and Functional Adaptation of Artificial Neural Networks.

ENN: A Neural Network with DCT Adaptive Activation Functions

Data-Specific Activation Function Learning for Hyperspectral Image Classification

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

A novel activation function for multilayer feed-forward neural networks

Deep Networks with Non-Static Activation Function

Adaptive Activation Functions for Predictive Modeling with Sparse Experimental Data

An Efficient Asymmetric Nonlinear Activation Function for Deep Neural Networks

Why KDAC? A general activation function for knowledge discovery

Adaptive Parametric Activation

A simple and efficient architecture for trainable activation functions

Diverse Activation Functions in Deep Learning

An Ann Modeling Method With Optimizing Net Structure And Activation Function Types

Activation function optimization method: Learnable series linear units (LSLUs)