Abstract:Effective activation functions introduce non-linear transformations, providing neural networks with stronger fitting capa-bilities, which help them better adapt to real data distributions. Huawei Noah's Lab believes that dynamic activation functions are more suitable than static activation functions for enhancing the non-linear capabilities of neural networks. Tsinghua University's related research also suggests using dynamically adjusted activation functions. Building on the ideas of using fine-tuned activation functions from Tsinghua University and Huawei Noah's Lab, we propose a series-based learnable ac-tivation function called LSLU (Learnable Series Linear Units). This method simplifies deep learning networks while im-proving accuracy. This method introduces learnable parameters {\theta} and {\omega} to control the activation function, adapting it to the current layer's training stage and improving the model's generalization. The principle is to increase non-linearity in each activation layer, boosting the network's overall non-linearity. We evaluate LSLU's performance on CIFAR10, CIFAR100, and specific task datasets (e.g., Silkworm), validating its effectiveness. The convergence behavior of the learnable parameters {\theta} and {\omega}, as well as their effects on generalization, are analyzed. Our empirical results show that LSLU enhances the general-ization ability of the original model in various tasks while speeding up training. In VanillaNet training, parameter {\theta} initially decreases, then increases before stabilizing, while {\omega} shows an opposite trend. Ultimately, LSLU achieves a 3.17% accuracy improvement on CIFAR100 for VanillaNet (Table 3). Codes are available at <a class="link-external link-https" href="https://github.com/vontran2021/Learnable-series-linear-units-LSLU" rel="external noopener nofollow">this https URL</a>.

Learning a Single Neuron for Non-monotonic Activation Functions

Agnostic Learning of Arbitrary ReLU Activation under Gaussian Marginals

Saturated Non-Monotonic Activation Functions

Learnable Surrogate Gradient for Direct Training Spiking Neural Networks

Convergence Analysis of Two-layer Neural Networks with ReLU Activation

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

On the Complexity of Learning Neural Networks

Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalization

SGD Learns One-Layer Networks in WGANs

Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuron

The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks

Expressive Monotonic Neural Networks

Activation function optimization method: Learnable series linear units (LSLUs)

Learning Narrow One-Hidden-Layer ReLU Networks

BrowNNe: Brownian Nonlocal Neurons & Activation Functions

Symmetric Single Index Learning

On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting

The role of optimization geometry in single neuron learning

Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions

The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Models

Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods