Abstract:RNN models have achieved the state-of-the-art performance in a wide range of text mining tasks. However, these models are often regarded as black-boxes and are criticized due to the lack of interpretability. In this paper, we enhance the interpretability of RNNs by providing interpretable rationales for RNN predictions. Nevertheless, interpreting RNNs is a challenging problem. Firstly, unlike existing methods that rely on local approximation, we aim to provide rationales that are more faithful to the decision making process of RNN models. Secondly, a flexible interpretation method should be able to assign contribution scores to text segments of varying lengths, instead of only to individual words. To tackle these challenges, we propose a novel attribution method, called REAT, to provide interpretations to RNN predictions. REAT decomposes the final prediction of a RNN into additive contribution of each word in the input text. This additive decomposition enables REAT to further obtain phrase-level attribution scores. In addition, REAT is generally applicable to various RNN architectures, including GRU, LSTM and their bidirectional versions. Experimental results demonstrate the faithfulness and interpretability of the proposed attribution method. Comprehensive analysis shows that our attribution method could unveil the useful linguistic knowledge captured by RNNs. Some analysis further demonstrates our method could be utilized as a debugging tool to examine the vulnerability and failure reasons of RNNs, which may lead to several promising future directions to promote generalization ability of RNNs.

On Attribution of Recurrent Neural Network Predictions via Additive Decomposition

Towards Interpreting Recurrent Neural Networks Through Probabilistic Abstraction

Residual Recurrent Neural Networks for Learning Sequential Representations.

Evaluating Recurrent Neural Network Explanations

Understanding Hidden Memories of Recurrent Neural Networks

A comprehensive and reliable feature attribution method: Double-sided remove and reconstruct (DoRaR)

Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networks

DeepSeer: Interactive RNN Explanation and Debugging via State Abstraction

On Discprecncies between Perturbation Evaluations of Graph Neural Network Attributions

MEME: Generating RNN Model Explanations via Model Extraction

Decomposing a Recurrent Neural Network into Modules for Enabling Reusability and Replacement

Increasing the Interpretability of Recurrent Neural Networks Using Hidden Markov Models

Asymmetric feature interaction for interpreting model predictions

An Interpretability Algorithm of Neural Network Based on Neural Support Decision Tree

DeepCover: Advancing RNN test coverage and online error prediction using state machine extraction

Towards Hierarchical Importance Attribution: Explaining Compositional Semantics for Neural Sequence Models

Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network.

Generating Hierarchical Explanations on Text Classification Without Connecting Rules

Interpretable Neural Predictions with Differentiable Binary Variables

Improving Interpretability of Deep Neural Networks with Semantic Information