Abstract:Deep Learning (DL) stands out as a leading model for processing high-dimensional data, where the nonlinear transformation of hidden layers effectively extracts features. However, these unexplainable features make DL a low interpretability model. Conversely, Bayesian network (BN) is transparent and highly interpretable, and it can be helpful for interpreting DL. To improve the interpretability of DL from the perspective of feature cognition, we propose the feature analysis network (FAN), a DL structure fused with BN. FAN retains the DL feature extraction capability and applies BN as the output layer to learn the relationships between the features and the outputs. These relationships can be probabilistically represented by the structure and parameters of the BN, intuitively. In a further study, a correlation clustering-based feature analysis network (cc-FAN) is proposed to detect the correlations among inputs and to preserve this information to explain the features' physical meaning to a certain extent. To quantitatively evaluate the interpretability of the model, we design the network simplification and interpretability indicators separately. Experiments on eight datasets show that FAN has better interpretability than that of the other models with basically unchanged model accuracy and similar model complexities. On the radar effect mechanism dataset, from the feature structure-based relevance interpretability indicator, FAN is up to 4.8 times better than that of the other models, and cc-FAN is up to 21.5 times better than that of the other models. FAN and cc-FAN enhance the interpretability of the DL model structure from the aspects of features; moreover, based on the input correlations, cc-FAN can help us to better understand the physical meaning of features.

Interpreting and Disentangling Feature Components of Various Complexity from DNNs

Understanding, Analyzing, and Optimizing the Complexity of Deep Models

Going Beyond Neural Network Feature Similarity: the Network Feature Complexity and Its Interpretation Using Category Theory

Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs.

Discovering and Explaining the Representation Bottleneck of DNNs

Understanding Visual Feature Reliance through the Lens of Complexity

Feature Analysis Network: An Interpretable Idea in Deep Learning

Understanding the Dynamics of DNNs Using Graph Modularity

Interpretability for Reliable, Efficient, and Self-Cognitive DNNs: from Theories to Applications.

Deciphering the Feature Representation of Deep Neural Networks for High-Performance AI

Towards the Difficulty for a Deep Neural Network to Learn Concepts of Different Complexities

Functional Network: A Novel Framework for Interpretability of Deep Neural Networks

Exploring Linear Feature Disentanglement for Neural Networks

Interpreting Hierarchical Linguistic Interactions in DNNs

A Measure of the Complexity of Neural Representations based on Partial Information Decomposition

Label-Based Disentanglement Measure among Hidden Units of Deep Learning

Neural Complexity Measures

Towards the Dynamics of a DNN Learning Symbolic Interactions

An Information Theoretic Interpretation to Deep Neural Networks.

A Geometric Modeling of Occam's Razor in Deep Learning