Abstract:Neural networks are powerful function approximators, yet their ``black-box" nature often renders them opaque and difficult to interpret. While many post-hoc explanation methods exist, they typically fail to capture the underlying reasoning processes of the networks. A truly interpretable neural network would be trained similarly to conventional models using techniques such as backpropagation, but additionally provide insights into the learned input-output relationships. In this work, we introduce the concept of interpretability pipelineing, to incorporate multiple interpretability techniques to outperform each individual technique. To this end, we first evaluate several architectures that promise such interpretability, with a particular focus on two recent models selected for their potential to incorporate interpretability into standard neural network architectures while still leveraging backpropagation: the Growing Interpretable Neural Network (GINN) and Kolmogorov Arnold Networks (KAN). We analyze the limitations and strengths of each and introduce a novel interpretable neural network GINN-KAN that synthesizes the advantages of both models. When tested on the Feynman symbolic regression benchmark datasets, GINN-KAN outperforms both GINN and KAN. To highlight the capabilities and the generalizability of this approach, we position GINN-KAN as an alternative to conventional black-box networks in Physics-Informed Neural Networks (PINNs). We expect this to have far-reaching implications in the application of deep learning pipelines in the natural sciences. Our experiments with this interpretable PINN on 15 different partial differential equations demonstrate that GINN-KAN augmented PINNs outperform PINNs with black-box networks in solving differential equations and surpass the capabilities of both GINN and KAN.

GPEX, A Framework For Interpreting Artificial Neural Networks

Towards Interpreting Recurrent Neural Networks Through Probabilistic Abstraction

GINN-KAN: Interpretability pipelining with applications in Physics Informed Neural Networks

Scalable Partial Explainability in Neural Networks via Flexible Activation Functions

Explainable Artificial Intelligence by Genetic Programming: A Survey

Interpret Gaussian Process Models by Using Integrated Gradients

GAMI-Net: An Explainable Neural Network based on Generalized Additive Models with Structured Interactions

Opening the Black Box of Neural Networks: Methods for Interpreting Neural Network Models in Clinical Applications

Explainable Learning with Gaussian Processes

ProtGNN: Towards Self-Explaining Graph Neural Networks.

Gnothi Seauton: Empowering Faithful Self-Interpretability in Black-Box Models

Neural network interpretability with layer-wise relevance propagation: novel techniques for neuron selection and visualization

Explaining Genetic Programming Trees using Large Language Models

Enhancing Interpretability in AI-Generated Image Detection with Genetic Programming

Gaussian Process Kolmogorov-Arnold Networks

Adaptive Explainable Neural Networks (Axnns)

How Interpretable Are Interpretable Graph Neural Networks?

GraphXAIN: Narratives to Explain Graph Neural Networks

Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond

The Intelligible and Effective Graph Neural Additive Networks

GNNExplainer: Generating Explanations for Graph Neural Networks