Abstract:A pivotal aspect in the design of neural networks lies in selecting activation functions, crucial for introducing nonlinear structures that capture intricate input-output patterns. While the effectiveness of adaptive or trainable activation functions has been studied in domains with ample data, like image classification problems, significant gaps persist in understanding their influence on classification accuracy and predictive uncertainty in settings characterized by limited data availability. This research aims to address these gaps by investigating the use of two types of adaptive activation functions. These functions incorporate shared and individual trainable parameters per hidden layer and are examined in three testbeds derived from additive manufacturing problems containing fewer than one hundred training instances. Our investigation reveals that adaptive activation functions, such as Exponential Linear Unit (ELU) and Softplus, with individual trainable parameters, result in accurate and confident prediction models that outperform fixed-shape activation functions and the less flexible method of using identical trainable activation functions in a hidden layer. Therefore, this work presents an elegant way of facilitating the design of adaptive neural networks in scientific and engineering problems.

Multi-Activation Hidden Units for Neural Networks with Random Weights

Efficient Design of Neural Networks with Random Weights

What's Hidden in a Randomly Weighted Neural Network?

Residual Random Neural Networks

Activation Ensembles for Deep Neural Networks

Neural Networks with Activation Networks

Unbiased Weight Maximization

A novel type of activation function in artificial neural networks: Trained activation function

Building Feedforward Neural Networks with Random Weights for Large Scale Datasets

Weights and Structure Determination of Multiple-Input Feed-Forward Neural Network Activated by Chebyshev Polynomials of Class 2 Via Cross-Validation

Learning Discrete Weights and Activations Using the Local Reparameterization Trick

Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks

Adaptive Activation Functions for Predictive Modeling with Sparse Experimental Data

Bayesian optimization for sparse neural networks with trainable activation functions

Generating Random Parameters in Feedforward Neural Networks with Random Hidden Nodes: Drawbacks of the Standard Method and How to Improve It

Neural Network with Multiple Connection Weights.

Adaptive Parametric Activation

Learning Combinations of Activation Functions

Latent Assistance Networks: Rediscovering Hyperbolic Tangents in RL

Evolving activation dynamics in feedforward neural networks

Neural Networks with Few Multiplications