Abstract:We study how well large language models (LLMs) explain their generations through rationales -- a set of tokens extracted from the input text that reflect the decision-making process of LLMs. Specifically, we systematically study rationales derived using two approaches: (1) popular prompting-based methods, where prompts are used to guide LLMs in generating rationales, and (2) technical attribution-based methods, which leverage attention or gradients to identify important tokens. Our analysis spans three classification datasets with annotated rationales, encompassing tasks with varying performance levels. While prompting-based self-explanations are widely used, our study reveals that these explanations are not always as "aligned" with the human rationale as attribution-based explanations. Even more so, fine-tuning LLMs to enhance classification task accuracy does not enhance the alignment of prompting-based rationales. Still, it does considerably improve the alignment of attribution-based methods (e.g., InputXGradient). More importantly, we show that prompting-based self-explanation is also less "faithful" than attribution-based explanations, failing to provide a reliable account of the model's decision-making process. To evaluate faithfulness, unlike prior studies that excluded misclassified examples, we evaluate all instances and also examine the impact of fine-tuning and accuracy on alignment and faithfulness. Our findings suggest that inconclusive faithfulness results reported in earlier studies may stem from low classification accuracy. These findings underscore the importance of more rigorous and comprehensive evaluations of LLM rationales.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to evaluate the performance of explanations (i.e., rationales) generated by large - language models (LLMs) in terms of human alignment and model faithfulness. Specifically, the researchers focus on two main aspects: 1. **Human Alignment**: This refers to the degree of match between the explanations generated by the model and those labeled by humans. By comparing the rationales generated by the model with the explanations provided by human experts, we can assess whether the model's explanations are in line with the human decision - making process. 2. **Model Faithfulness**: This refers to whether the explanations generated by the model truly reflect its internal decision - making process. By masking the words that the model considers important (i.e., rationales) and observing the changes in the model's predictions, we can assess whether these explanations have an actual impact on the model's decision - making. To comprehensively evaluate these two aspects, the researchers adopted two different methods to extract rationales: - **Prompting - Based Methods**: Guide the model to generate explanations by designing specific prompts. - **Attribution - Based Methods**: Use techniques such as attention weights or gradients to identify important words in the input text. The study also explored the impact of fine - tuning on model performance, especially how to improve the accuracy of classification tasks through fine - tuning and further affect the human alignment and model faithfulness of rationales. The experimental results show that, in most cases, the attribution - based methods are more in line with human explanations than the prompting - based methods and show higher faithfulness after fine - tuning. In addition, the study found that low classification performance and collapsing predictions are an important limiting factor in evaluating model faithfulness, which highlights the need to consider task and model dependence when evaluating the explanatory ability of LLMs. In summary, this study provides a systematic framework for extracting and evaluating the rationales of LLMs and reveals the key limitations of prompting - based rationales in terms of explanatory ability and credibility.

Evaluating Human Alignment and Model Faithfulness of LLM Rationale

Are self-explanations from Large Language Models faithful?

FaithLM: Towards Faithful Explanations for Large Language Models

Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models

Local Explanations and Self-Explanations for Assessing Faithfulness in black-box LLMs

Towards Faithful Natural Language Explanations: A Study Using Activation Patching in Large Language Models

Post Hoc Explanations of Language Models Can Improve Language Models

Dissociation of Faithful and Unfaithful Reasoning in LLMs

Why Would You Suggest That? Human Trust in Language Model Responses

Enhancing Large Language Models' Situated Faithfulness to External Contexts

Properties and Challenges of LLM-Generated Explanations

On Measuring Faithfulness or Self-consistency of Natural Language Explanations

Aligning Large Language Models for Faithful Integrity Against Opposing Argument

Beyond Labels: Aligning Large Language Models with Human-like Reasoning

Evaluating the Reliability of Self-Explanations in Large Language Models

Alignment Between the Decision-Making Logic of LLMs and Human Cognition: A Case Study on Legal LLMs

Large Language Models as Evaluators for Recommendation Explanations

On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models

Evaluating Consistency and Reasoning Capabilities of Large Language Models

Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Models

Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationales