Abstract:Transformer-based language models have been shown to be highly effective for several NLP tasks. In this paper, we consider three transformer models, BERT, RoBERTa, and XLNet, in both small and large versions, and investigate how faithful their representations are with respect to the semantic content of texts. We formalize a notion of semantic faithfulness, in which the semantic content of a text should causally figure in a model's inferences in question answering. We then test this notion by observing a model's behavior on answering questions about a story after performing two novel semantic interventions: deletion intervention and negation intervention. While transformer models achieve high performance on standard question answering tasks, we show that they fail to be semantically faithful once we perform these interventions for a significant number of cases (~50% for deletion intervention, and ~20% drop in accuracy for negation intervention). We then propose an intervention-based training regime that can mitigate the undesirable effects for deletion intervention by a significant margin (from ~ 50% to ~6%). We analyze the inner-workings of the models to better understand the effectiveness of intervention-based training for deletion intervention. But we show that this training does not attenuate other aspects of semantic unfaithfulness such as the models' inability to deal with negation intervention or to capture the predicate-argument structure of texts. We also test InstructGPT, via prompting, for its ability to handle the two interventions and to capture predicate-argument structure. While InstructGPT models do achieve very high performance on predicate-argument structure task, they fail to respond adequately to our deletion and negation interventions.

Developmental Negation Processing in Transformer Language Models

How Transformers Solve Propositional Logic Problems: A Mechanistic Analysis

Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models

Reasoning in Transformers -- Mitigating Spurious Correlations and Reasoning Shortcuts

A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task

Transformer-based Language Models for Reasoning in the Description Logic ALCQ

Teaching Probabilistic Logical Reasoning to Transformers

Mechanisms of Symbol Processing for In-Context Learning in Transformer Networks

On the Ability and Limitations of Transformers to Recognize Formal Languages

Analyzing Semantic Faithfulness of Language Models via Input Intervention on Question Answering

The Antecedents of Transformer Models

Can Transformers Reason Logically? A Study in SAT Solving

When can transformers reason with abstract symbols?

Transformer Models for Type Inference in the Simply Typed Lambda Calculus: A Case Study in Deep Learning for Code

Transformers as Soft Reasoners over Language

Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics

Estimating the Causal Effects of Natural Logic Features in Transformer-Based NLI Models

Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge

How do we get there? Evaluating transformer neural networks as cognitive models for English past tense inflection

Faith and Fate: Limits of Transformers on Compositionality

Understanding by Understanding Not: Modeling Negation in Language Models