Abstract:This paper introduces KnowHalu, a novel approach for detecting hallucinations in text generated by large language models (LLMs), utilizing step-wise reasoning, multi-formulation query, multi-form knowledge for factual checking, and fusion-based detection mechanism. As LLMs are increasingly applied across various domains, ensuring that their outputs are not hallucinated is critical. Recognizing the limitations of existing approaches that either rely on the self-consistency check of LLMs or perform post-hoc fact-checking without considering the complexity of queries or the form of knowledge, KnowHalu proposes a two-phase process for hallucination detection. In the first phase, it identifies non-fabrication hallucinations--responses that, while factually correct, are irrelevant or non-specific to the query. The second phase, multi-form based factual checking, contains five key steps: reasoning and query decomposition, knowledge retrieval, knowledge optimization, judgment generation, and judgment aggregation. Our extensive evaluations demonstrate that KnowHalu significantly outperforms SOTA baselines in detecting hallucinations across diverse tasks, e.g., improving by 15.65% in QA tasks and 5.50% in summarization tasks, highlighting its effectiveness and versatility in detecting hallucinations in LLM-generated content.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the "hallucination" problem that occurs when large language models (LLMs) generate text. Specifically, these hallucinations can be divided into two categories: 1. **Non - fabrication hallucinations**: Although these hallucinations are factually correct, they are either irrelevant to the question or not specific. For example, when asked "What is the main language in Barcelona?", answering "European languages" is correct but not specific. 2. **Fabrication hallucinations**: These hallucinations contain false factual information or assumptions. For example, when asked "When was 'The Sound of Silence' released?", answering "1966" is wrong; the correct answer should be 1964. Existing hallucination detection methods mainly rely on the self - consistency check of LLMs or post - processing fact - checking, but these methods have limitations, such as not considering the complexity of the query and the form of knowledge. Therefore, this paper proposes a new multi - stage hallucination detection mechanism - **KnowHalu**, aiming to more effectively detect these two types of hallucinations through fact - checking of multi - form knowledge. ### Main contributions 1. **Two - stage detection framework**: - **Non - fabrication hallucination detection**: Check the specificity of the answer by extracting tasks to identify hallucinations that are factually correct but irrelevant to the question. - **Fact - checking**: Conduct detailed fact - checking through five steps (step - by - step reasoning and querying, knowledge retrieval, knowledge optimization, judgment based on multi - form knowledge, aggregation). 2. **Utilization of multi - form knowledge**: - **Unstructured knowledge**: Relevant paragraphs retrieved from the knowledge base. - **Structured knowledge**: Knowledge represented in the form of object - predicate - object triples. 3. **Step - by - step reasoning and querying**: - Decompose the original query into multiple sub - queries, and conduct detailed fact - checking for each sub - query to ensure the accuracy of each step. 4. **Knowledge retrieval and optimization**: - Use the Retrieval - Augmented Generation (RAG) framework to retrieve relevant knowledge from knowledge bases such as Wikipedia, and optimize the knowledge through LLM to make it more concise and useful. 5. **Judgment and aggregation based on multi - form knowledge**: - Evaluate each sub - query and its corresponding knowledge through LLM to generate a preliminary judgment. - Aggregate the judgment results of different forms of knowledge to improve the accuracy of the final judgment. ### Experimental results The experimental results show that KnowHalu significantly outperforms the existing state - of - the - art methods (SOTA) in multiple tasks. For example, it improves by 15.65% in question - answering tasks and 5.50% in summarization tasks. This indicates that KnowHalu not only has higher accuracy in detecting hallucinations but also can handle complex queries and logic. ### Summary KnowHalu effectively solves the hallucination problem when LLMs generate text, especially non - fabrication hallucinations and fabrication hallucinations, by introducing a multi - stage detection framework and fact - checking of multi - form knowledge. This method not only improves the accuracy of hallucination detection but also enhances the model's ability to handle complex queries.

KnowHalu: Hallucination Detection via Multi-Form Knowledge Based Factual Checking

Knowledge-Centric Hallucination Detection

Cost-Effective Hallucination Detection for LLMs

MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Models

The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models

Halu-J: Critique-Based Hallucination Judge

Hallucination Detection and Hallucination Mitigation: An Investigation

Fine-grained Hallucination Detection and Mitigation in Long-form Question Answering

Medico: Towards Hallucination Detection and Correction with Multi-source Evidence Fusion

DelucionQA: Detecting Hallucinations in Domain-specific Question Answering

Developing a Reliable, General-Purpose Hallucination Detection and Mitigation Service: Insights and Lessons Learned

Fine-grained Hallucination Detection and Editing for Language Models

FactCHD: Benchmarking Fact-Conflicting Hallucination Detection

Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus

Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modeling

FactCheckmate: Preemptively Detecting and Mitigating Hallucinations in LMs

HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models

Drowzee: Metamorphic Testing for Fact-Conflicting Hallucination Detection in Large Language Models

Knowledge Verification to Nip Hallucination in the Bud

Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models

Towards Detecting LLMs Hallucination via Markov Chain-based Multi-agent Debate Framework