Abstract:In the past, research on a single low dimensional activation function in networks has led to internal covariate shift and gradient deviation problems. A relatively small research area is how to use function combinations to provide property completion for a single activation function application. We propose a network adversarial method to address the aforementioned challenges. This is the first method to use different activation functions in a network. Based on the existing activation functions in the current network, an adversarial function with opposite derivative image properties is constructed, and the two are alternately used as activation functions for different network layers. For complex situations, we propose a method of high-dimensional function graph decomposition(HD-FGD), which divides it into different parts and then passes through a linear layer. After integrating the inverse of the partial derivatives of each decomposed term, we obtain its adversarial function by referring to the computational rules of the decomposition process. The use of network adversarial methods or the use of HD-FGD alone can effectively replace the traditional MLP+activation function mode. Through the above methods, we have achieved a substantial improvement over standard activation functions regarding both training efficiency and predictive accuracy. The article addresses the adversarial issues associated with several prevalent activation functions, presenting alternatives that can be seamlessly integrated into existing models without any adverse effects. We will release the code as open source after the conference review process is completed.

Conditional Activation for Diverse Neurons in Heterogeneous Networks

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

Activity-dependent Neuron Model for Noise Resistance

Activation Space Selectable Kolmogorov-Arnold Networks

Conditional computation in neural networks: principles and research trends

Neuronal diversity can improve machine learning for physics and beyond

Dynamics of heterogeneous Hopfield neural network with adaptive activation function based on memristor

Neural Networks with Activation Networks

A New Artificial Neuron Proposal with Trainable Simultaneous Local and Global Activation Function

Decoding Spiking Mechanism with Dynamic Learning on Neuron Population

Biologically Inspired Heterogeneous Learning for Accurate, Efficient and Low-Latency Neural Network

Learning Combinations of Activation Functions

Neural networks simulating short-term memory of two inputs with varying commonality

Latent Assistance Networks: Rediscovering Hyperbolic Tangents in RL

Your Network May Need to Be Rewritten: Network Adversarial Based on High-Dimensional Function Graph Decomposition

Coexisting firing patterns and attractor selection in memristive synapse coupled heterogeneous neurons

Single-unit activations confer inductive biases for emergent circuit solutions to cognitive tasks

Salience-Affected Neural Networks

Periodic dynamics of multidirectional associative neural networks with discontinuous activation functions and mixed time delays

Neuronal heterogeneity of normalization strength in a circuit model

Regularized Flexible Activation Function Combinations for Deep Neural Networks