Abstract:Although many studies have investigated and reduced hallucinations in large language models (LLMs) for single-document tasks, research on hallucination in multi-document summarization (MDS) tasks remains largely unexplored. Specifically, it is unclear how the challenges arising from handling multiple documents (e.g., repetition and diversity of information) affect models outputs. In this work, we investigate how hallucinations manifest in LLMs when summarizing topic-specific information from multiple documents. Since no benchmarks exist for investigating hallucinations in MDS, we use existing news and conversation datasets, annotated with topic-specific insights, to create two novel multi-document benchmarks. When evaluating 5 LLMs on our benchmarks, we observe that on average, up to 75% of the content in LLM-generated summary is hallucinated, with hallucinations more likely to occur towards the end of the summaries. Moreover, when summarizing non-existent topic-related information, gpt-3.5-turbo and GPT-4o still generate summaries about 79.35% and 44% of the time, raising concerns about their tendency to fabricate content. To understand the characteristics of these hallucinations, we manually evaluate 700+ insights and find that most errors stem from either failing to follow instructions or producing overly generic insights. Motivated by these observations, we investigate the efficacy of simple post-hoc baselines in mitigating hallucinations but find them only moderately effective. Our results underscore the need for more effective approaches to systematically mitigate hallucinations in MDS. We release our dataset and code at <a class="link-external link-http" href="http://github.com/megagonlabs/Hallucination_MDS" rel="external noopener nofollow">this http URL</a>.

SemEval-2024 Shared Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes

AILS-NTUA at SemEval-2024 Task 6: Efficient model tuning for hallucination detection and analysis

SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-Based Classification for Hallucination Detection

MALTO at SemEval-2024 Task 6: Leveraging Synthetic Data for LLM Hallucination Detection

SLPL SHROOM at SemEval2024 Task 06: A comprehensive study on models ability to detect hallucination

SmurfCat at SemEval-2024 Task 6: Leveraging Synthetic Data for Hallucination Detection

MetaCheckGPT -- A Multi-task Hallucination Detector Using LLM Uncertainty and Meta-models

Survey of Hallucination in Natural Language Generation

HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wild

THaMES: An End-to-End Tool for Hallucination Mitigation and Evaluation in Large Language Models

HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation

Zero-Shot Multi-task Hallucination Detection

HypoTermQA: Hypothetical Terms Dataset for Benchmarking Hallucination Tendency of LLMs

The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models

HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models

The Two Sides of the Coin: Hallucination Generation and Detection with LLMs as Evaluators for LLMs

From Single to Multi: How LLMs Hallucinate in Multi-Document Summarization

Hallucination Detection and Hallucination Mitigation: An Investigation

OPDAI at SemEval-2024 Task 6: Small LLMs can Accelerate Hallucination Detection with Weakly Supervised Data

Developing a Reliable, General-Purpose Hallucination Detection and Mitigation Service: Insights and Lessons Learned

Mitigating Open-Vocabulary Caption Hallucinations