Abstract:Deep Learning (DL) stands out as a leading model for processing high-dimensional data, where the nonlinear transformation of hidden layers effectively extracts features. However, these unexplainable features make DL a low interpretability model. Conversely, Bayesian network (BN) is transparent and highly interpretable, and it can be helpful for interpreting DL. To improve the interpretability of DL from the perspective of feature cognition, we propose the feature analysis network (FAN), a DL structure fused with BN. FAN retains the DL feature extraction capability and applies BN as the output layer to learn the relationships between the features and the outputs. These relationships can be probabilistically represented by the structure and parameters of the BN, intuitively. In a further study, a correlation clustering-based feature analysis network (cc-FAN) is proposed to detect the correlations among inputs and to preserve this information to explain the features' physical meaning to a certain extent. To quantitatively evaluate the interpretability of the model, we design the network simplification and interpretability indicators separately. Experiments on eight datasets show that FAN has better interpretability than that of the other models with basically unchanged model accuracy and similar model complexities. On the radar effect mechanism dataset, from the feature structure-based relevance interpretability indicator, FAN is up to 4.8 times better than that of the other models, and cc-FAN is up to 21.5 times better than that of the other models. FAN and cc-FAN enhance the interpretability of the DL model structure from the aspects of features; moreover, based on the input correlations, cc-FAN can help us to better understand the physical meaning of features.

Exploring Linear Feature Disentanglement for Neural Networks

Multi-level Disentanglement Graph Neural Network

Understanding Deep Neural Networks via Linear Separability of Hidden Layers

Interpretable Disentanglement of Neural Networks by Extracting Class-Specific Subnetwork

Half-Space Feature Learning in Neural Networks

Learning nonlinear manifolds based on mixtures of localized linear manifolds under a self-organizing framework

Functional Network: A Novel Framework for Interpretability of Deep Neural Networks

Going Beyond Linear Mode Connectivity: the Layerwise Linear Feature Connectivity.

Interpret Neural Networks by Extracting Critical Subnetworks

Disentangled Non-Local Neural Networks

Randomly Initialized One-Layer Neural Networks Make Data Linearly Separable

Visualising Feature Learning in Deep Neural Networks by Diagonalizing the Forward Feature Map

How to Explain Neural Networks: A perspective of data space division

Linear Laplacian Discrimination for Feature Extraction

Local Interpretation of Transformer Based on Linear Decomposition

Feature Analysis Network: An Interpretable Idea in Deep Learning

Dissecting Deep Neural Networks

Adversarial Noises Are Linearly Separable for (Nearly) Random Neural Networks

Dissecting the Diffusion Process in Linear Graph Convolutional Networks

Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs.

Feature Extraction by Maximizing the Average Neighborhood Margin