Abstract:Large Language Models (LLMs) are often augmented with external information as contexts, but this external information can sometimes be inaccurate or even intentionally misleading. We argue that robust LLMs should demonstrate situated faithfulness, dynamically calibrating their trust in external information based on their confidence in the internal knowledge and the external context. To benchmark this capability, we evaluate LLMs across several QA datasets, including a newly created dataset called RedditQA featuring in-the-wild incorrect contexts sourced from Reddit posts. We show that when provided with both correct and incorrect contexts, both open-source and proprietary models tend to overly rely on external information, regardless of its factual accuracy. To enhance situated faithfulness, we propose two approaches: Self-Guided Confidence Reasoning (SCR) and Rule-Based Confidence Reasoning (RCR). SCR enables models to self-access the confidence of external information relative to their own internal knowledge to produce the most accurate answer. RCR, in contrast, extracts explicit confidence signals from the LLM and determines the final answer using predefined rules. Our results show that for LLMs with strong reasoning capabilities, such as GPT-4o and GPT-4o mini, SCR outperforms RCR, achieving improvements of up to 24.2% over a direct input augmentation baseline. Conversely, for a smaller model like Llama-3-8B, RCR outperforms SCR. Fine-tuning SCR with our proposed Confidence Reasoning Direct Preference Optimization (CR-DPO) method improves performance on both seen and unseen datasets, yielding an average improvement of 8.9% on Llama-3-8B. In addition to quantitative results, we offer insights into the relative strengths of SCR and RCR. Our findings highlight promising avenues for improving situated faithfulness in LLMs. The data and code are released.

Large Language Models with Controllable Working Memory

Supervised Knowledge Makes Large Language Models Better In-context Learners

Large Language Models and Knowledge Graphs: Opportunities and Challenges

MEMORYLLM: Towards Self-Updatable Large Language Models

Can Language Models Act as Knowledge Bases at Scale?

When Context Leads but Parametric Memory Follows in Large Language Models

MindLLM: Pre-training Lightweight Large Language Model from Scratch, Evaluations and Domain Applications

A Principled Framework for Knowledge-enhanced Large Language Model

Large Knowledge Model: Perspectives and Challenges

MindLLM: Lightweight Large Language Model Pre-Training, Evaluation and Domain Application

Towards Concept-Aware Large Language Models

Enhancing Large Language Model with Self-Controlled Memory Framework

Introspective Tips: Large Language Model for In-Context Decision Making

Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts

Large Language Models Can Self-Improve in Long-context Reasoning

Updating Large Language Models’ Memories with Time Constraints

CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory

Enhancing Large Language Models' Situated Faithfulness to External Contexts

Challenges and Contributing Factors in the Utilization of Large Language Models (LLMs)

Are Large Language Models Really Robust to Word-Level Perturbations?