Abstract:Disseminating and incorporating logic rules into deep neural networks has been extensively explored for sentiment classification in recent years. In particular, most methods and algorithms proposed for this purpose rely on a specific component that aims to capture and model logic rules, followed by a sequence model to process the input sequence. While the authors of these methods claim that they effectively capture syntactic structures that affect sentiment classification, they only show improvement in accuracy to support their claims without further analysis. Focusing on various syntactic structures, particularly contrastive discourse relations such as the A-but-B structure, we introduce the PERCY score, a novel Post-hoc Explanation-based Rule ConsistencY Score to analyze and study the ability of several of these methods to identify these structures in a given sentence, and to make their classification decisions based on the appropriate conjunct. Specifically, we explore the use of model-agnostic post-hoc explanation frameworks to explain the predictions of any classifier in an interpretable and faithful manner. These model explainability frameworks provide feature attribution scores to estimate each word's impact on the final classification decision. Then, they are combined to check whether the model has based its decision on the right conjunct. Our experiments show that (a) accuracy – or any other performance metric – can be misleading in assessing the ability of logic rule dissemination methods to base their decisions on the right conjunct, (b) not all analyzed methods effectively capture syntactic structures, (c) often, the underlying sequence model is what captures the structure, and (d) for the best method, less than 25% of the test examples are classified based on the appropriate conjunct, indicating that a lot of research needs to be done on this topic. Finally, we experimentally demonstrate that the PERCY scores calculated are robust and stable w.r.t. the feature-attribution frameworks used.

Efficiently Learning Probabilistic Logical Models by Cheaply Ranking Mined Rules

Concise and Organized Perception Facilitates Large Language Models for Deductive Reasoning.

Distilling Task-specific Logical Rules from Large Pre-trained Models

RNNLogic: Learning Logic Rules for Reasoning on Knowledge Graphs

Numerical Markov Logic Network: A Scalable Probabilistic Framework for Hybrid Knowledge Inference

LATENTLOGIC: Learning Logic Rules in Latent Space over Knowledge Graphs.

Logical Credal Networks

RLogic: Recursive Logical Rule Learning from Knowledge Graphs

Logic Rules as Explanations for Legal Case Retrieval

Beyond Markov Logic: Efficient Mining of Prediction Rules in Large Graphs

Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs

A Probabilistic Soft Logic Reasoning Model with Automatic Rule Learning.

ChatRule: Mining Logical Rules with Large Language Models for Knowledge Graph Reasoning

Explaining Point Processes by Learning Interpretable Temporal Logic Rules

Neuro-Symbolic Rule Lists

Logical Entity Representation in Knowledge-Graphs for Differentiable Rule Learning

Scalable Rule Lists Learning with Sampling

Probabilistic Logic Neural Networks for Reasoning

PERCY: A post-hoc explanation-based score for logic rule dissemination consistency assessment in sentiment classification

Inductive Logic Boosting

Advances in Probabilistic Logic Model and Learning