Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Facts

Baolong Bi,Shenghua Liu,Lingrui Mei,Yiwei Wang,Pengliang Ji,Xueqi Cheng

2024-05-21

Abstract:The knowledge within large language models (LLMs) may become outdated quickly. While in-context editing (ICE) is currently the most effective method for knowledge editing (KE), it is constrained by the black-box modeling of LLMs and thus lacks interpretability. Our work aims to elucidate the superior performance of ICE on the KE by analyzing the impacts of in-context new knowledge on token-wise distributions. We observe that despite a significant boost in logits of the new knowledge, the performance of is still hindered by stubborn knowledge. Stubborn knowledge refers to as facts that have gained excessive confidence during pretraining, making it hard to edit effectively. To address this issue and further enhance the performance of ICE, we propose a novel approach termed $\textbf{De}$coding by $\textbf{C}$ontrasting $\textbf{K}$nowledge (DeCK). DeCK derives the distribution of the next token by contrasting the logits obtained from the newly edited knowledge guided by ICE with those from the unedited parametric knowledge. Our experiments consistently demonstrate that DeCK enhances the confidence of LLMs in edited facts. For instance, it improves the performance of LLaMA3-8B-instruct on MQuAKE by up to 219%, demonstrating its capability to strengthen ICE in the editing of stubborn knowledge. Our work paves the way to develop the both effective and accountable KE methods for LLMs. (The source code is available at:

Computation and Language

What problem does this paper attempt to address?

The paper attempts to address the challenges encountered when updating knowledge in large language models (LLMs), particularly the difficulties in dealing with "stubborn knowledge." Specifically:

1. **Limitations of existing methods**: The most effective current knowledge editing (KE) method is In-Context Editing (ICE), but due to the black-box nature of LLMs, this method lacks interpretability and performs poorly when modifying certain strongly pre-trained knowledge.

2. **Stubborn knowledge problem**: LLMs that have undergone extensive pre-training have extremely high confidence in certain facts, making these facts difficult to edit effectively through external context prompts.

To address these issues, the authors propose the Decoding by Contrasting Knowledge (DeCK) method, which aims to enhance LLMs' confidence in post-editing facts by contrasting the logical distribution of newly edited knowledge with the original parameter knowledge. Experimental results show that DeCK can significantly improve the performance of LLMs in editing stubborn knowledge.