Abstract:LLM hallucination, where LLMs occasionally generate unfaithful text, poses significant challenges for their practical applications. Most existing detection methods rely on external knowledge, LLM fine-tuning, or hallucination-labeled datasets, and they do not distinguish between different types of hallucinations, which are crucial for improving detection performance. We introduce a new task, Hallucination Reasoning, which classifies LLM-generated text into one of three categories: aligned, misaligned, and fabricated. Our novel zero-shot method assesses whether LLM has enough knowledge about a given prompt and text. Our experiments conducted on new datasets demonstrate the effectiveness of our method in hallucination reasoning and underscore its importance for enhancing detection performance.

What problem does this paper attempt to address?

This paper attempts to solve the hallucination problem that occurs when large - language models (LLMs) generate text. Specifically, LLMs occasionally produce inaccurate or unverifiable content, which poses a significant challenge to their practical applications. Most of the existing detection methods rely on external knowledge, LLM fine - tuning, or hallucination - annotated datasets, and these methods fail to distinguish different types of hallucinations, which is crucial for improving detection performance. To this end, the author introduces a new task - Hallucination Reasoning, whose purpose is to classify the text generated by LLMs into one of the following three categories: - **Aligned**: The text is consistent with the knowledge of the LLM. - **Misaligned**: The text is inconsistent with the knowledge of the LLM due to random sampling or dependence on previous tokens. - **Fabricated**: The text generated by the LLM when it lacks relevant knowledge. The author proposes a novel zero - sample method to evaluate whether the LLM has sufficient knowledge for a given prompt and text. The experimental results show that this method is effective in hallucination reasoning on the new dataset and emphasize its importance for enhancing detection performance. ### Main Contributions 1. **New Hallucination Reasoning Task**: To better understand and detect hallucinations. 2. **MKT (Model Knowledge Test) Method**: A zero - sample method that does not require external knowledge, annotated datasets, or LLM fine - tuning, used to identify whether the LLM has sufficient knowledge for the prompt and text. 3. **Experimental Verification**: Demonstrates the superiority of this method in question - answering and free - form text generation, and proves that integrating it into existing detection algorithms can significantly improve performance. Through these contributions, the author aims to fill the gaps in existing methods, provide more accurate means of hallucination detection, and provide a new direction for further research on hallucination reasoning.

LLM Hallucination Reasoning with Zero-shot Knowledge Test

InterrogateLLM: Zero-Resource Hallucination Detection in LLM-Generated Answers

Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden States

Cost-Effective Hallucination Detection for LLMs

OPDAI at SemEval-2024 Task 6: Small LLMs can Accelerate Hallucination Detection with Weakly Supervised Data

KnowHalu: Hallucination Detection via Multi-Form Knowledge Based Factual Checking

A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection.

SLM Meets LLM: Balancing Latency, Interpretability and Consistency in Hallucination Detection

A Multiple-Fill-in-the-Blank Exam Approach for Enhancing Zero-Resource Hallucination Detection in Large Language Models

LLM Internal States Reveal Hallucination Risk Faced With a Query

Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigation

Distinguishing Ignorance from Error in LLM Hallucinations

The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models

INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection

Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models

A Debate-Driven Experiment on LLM Hallucinations and Accuracy

Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modeling

Chainpoll: A high efficacy method for LLM hallucination detection

A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection

SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-Based Classification for Hallucination Detection

Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models