Abstract:Achieving transparency in black-box deep learning algorithms is still an open challenge. High dimensional features and decisions given by deep neural networks (NN) require new algorithms and methods to expose its mechanisms. Current state-of-the-art NN interpretation methods (e.g. Saliency maps, DeepLIFT, LIME, etc.) focus more on the direct relationship between NN outputs and inputs rather than the NN structure and operations itself. In current deep NN operations, there is uncertainty over the exact role played by neurons with fixed activation functions. In this paper, we achieve partially explainable learning model by symbolically explaining the role of activation functions (AF) under a scalable topology. This is carried out by modeling the AFs as adaptive Gaussian Processes (GP), which sit within a novel scalable NN topology, based on the Kolmogorov-Arnold Superposition Theorem (KST). In this scalable NN architecture, the AFs are generated by GP interpolation between control points and can thus be tuned during the back-propagation procedure via gradient descent. The control points act as the core enabler to both local and global adjustability of AF, where the GP interpolation constrains the intrinsic autocorrelation to avoid over-fitting. We show that there exists a trade-off between the NN's expressive power and interpretation complexity, under linear KST topology scaling. To demonstrate this, we perform a case study on a binary classification dataset of banknote authentication. By quantitatively and qualitatively investigating the mapping relationship between inputs and output, our explainable model can provide interpretation over each of the one-dimensional attributes. These early results suggest that our model has the potential to act as the final interpretation layer for deep neural networks.

ENN: A Neural Network with DCT Adaptive Activation Functions

ENN: A Neural Network with DCT Adaptive Activation Functions

Activation Ensembles for Deep Neural Networks

Trainable Highly-expressive Activation Functions

An Efficient Asymmetric Nonlinear Activation Function for Deep Neural Networks

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

Ensemble of Convolutional Neural Networks Trained with Different Activation Functions

Improved Expressivity Through Dendritic Neural Networks.

EIS - Efficient and Trainable Activation Functions for Better Accuracy and Performance

Neural Networks with Activation Networks

Ensemble Neural Networks (ENN): A gradient-free stochastic method

Data-specific activation function learning for constructive neural networks

Adaptive Activation Functions for Predictive Modeling with Sparse Experimental Data

Improving the Expressive Power of Deep Neural Networks through Integral Activation Transform

Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks

Learning Compact Neural Networks Using Ordinary Differential Equations as Activation Functions

Scalable Partial Explainability in Neural Networks via Flexible Activation Functions

Learning continuous piecewise non-linear activation functions for deep neural networks

Arbitrary sensitive transitions in recurrent neural networks

Neuro-Inspired Deep Neural Networks with Sparse, Strong Activations

The Expressivity and Training of Deep Neural Networks: Toward the Edge of Chaos?