Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models

Yuji Zhang,Sha Li,Jiateng Liu,Pengfei Yu,Yi R. Fung,Jing Li,Manling Li,Heng Ji

2024-07-11

Abstract:Hallucination is often regarded as a major impediment for using large language models (LLMs), especially for knowledge-intensive tasks. Even when the training corpus consists solely of true statements, language models still generate hallucinations in the form of amalgamations of multiple facts. We coin this phenomenon as ``knowledge overshadowing'': when we query knowledge from a language model with multiple conditions, some conditions overshadow others, leading to hallucinated outputs. This phenomenon partially stems from training data imbalance, which we verify on both pretrained models and fine-tuned models, over a wide range of LM model families and sizes.From a theoretical point of view, knowledge overshadowing can be interpreted as over-generalization of the dominant conditions (patterns). We show that the hallucination rate grows with both the imbalance ratio (between the popular and unpopular condition) and the length of dominant condition description, consistent with our derived generalization bound. Finally, we propose to utilize overshadowing conditions as a signal to catch hallucination before it is produced, along with a training-free self-contrastive decoding method to alleviate hallucination during inference. Our proposed approach showcases up to 82% F1 for hallucination anticipation and 11.2% to 39.4% hallucination control, with different models and datasets.

Computation and Language

What problem does this paper attempt to address?

The paper primarily explores a critical issue in large language models (LLMs) known as knowledge overshadowing, which can lead to the generation of inaccurate or non-factual statements, referred to as "hallucinations." Specifically, when extracting knowledge with multiple conditions from a language model, some conditions may overshadow others, resulting in erroneous or illogical content. For example, when inquiring about female researchers in the field of deep learning, the model might incorrectly nominate male researcher Yoshua Bengio.

The paper validates this phenomenon through theoretical analysis and experiments, and presents two main contributions:
1. **Phenomenon Discovery and Quantification**: The authors discovered the knowledge overshadowing phenomenon in language models of different scales and quantified the relationship between data imbalance ratio, condition length, and hallucination rate.
2. **Solution**: They proposed a training-free self-contrastive decoding method to detect and mitigate hallucinations during the inference process. Experimental results show that this method can significantly reduce the hallucination rate, by up to 39.4%.

Overall, this paper aims to address the issue of hallucinations in large language models when handling multi-condition queries and provides an effective solution.