Abstract:Large Language Models (LLMs) are often augmented with external information as contexts, but this external information can sometimes be inaccurate or even intentionally misleading. We argue that robust LLMs should demonstrate situated faithfulness, dynamically calibrating their trust in external information based on their confidence in the internal knowledge and the external context. To benchmark this capability, we evaluate LLMs across several QA datasets, including a newly created dataset called RedditQA featuring in-the-wild incorrect contexts sourced from Reddit posts. We show that when provided with both correct and incorrect contexts, both open-source and proprietary models tend to overly rely on external information, regardless of its factual accuracy. To enhance situated faithfulness, we propose two approaches: Self-Guided Confidence Reasoning (SCR) and Rule-Based Confidence Reasoning (RCR). SCR enables models to self-access the confidence of external information relative to their own internal knowledge to produce the most accurate answer. RCR, in contrast, extracts explicit confidence signals from the LLM and determines the final answer using predefined rules. Our results show that for LLMs with strong reasoning capabilities, such as GPT-4o and GPT-4o mini, SCR outperforms RCR, achieving improvements of up to 24.2% over a direct input augmentation baseline. Conversely, for a smaller model like Llama-3-8B, RCR outperforms SCR. Fine-tuning SCR with our proposed Confidence Reasoning Direct Preference Optimization (CR-DPO) method improves performance on both seen and unseen datasets, yielding an average improvement of 8.9% on Llama-3-8B. In addition to quantitative results, we offer insights into the relative strengths of SCR and RCR. Our findings highlight promising avenues for improving situated faithfulness in LLMs. The data and code are released.

Aligning Large Language Models for Faithful Integrity Against Opposing Argument

Enhancing Large Language Models' Situated Faithfulness to External Contexts

Making Large Language Models Better Reasoners with Alignment

When to Trust LLMs: Aligning Confidence with Response Quality

FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"

FaithLM: Towards Faithful Explanations for Large Language Models

IterAlign: Iterative Constitutional Alignment of Large Language Models

Evaluating Human Alignment and Model Faithfulness of LLM Rationale

ABC Align: Large Language Model Alignment for Safety & Accuracy

Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models

Improving and Assessing the Fidelity of Large Language Models Alignment to Online Communities

Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate

Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment

Dissociation of Faithful and Unfaithful Reasoning in LLMs

Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMs

Aligning with Logic: Measuring, Evaluating and Improving Logical Consistency in Large Language Models

Large Language Model Alignment: A Survey

Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM

Towards Faithful Chain-of-Thought: Large Language Models are Bridging Reasoners

On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models

Beyond Labels: Aligning Large Language Models with Human-like Reasoning