Abstract:Recently, Multimodal Large Language Models (MLLMs) have made significant progress in the video comprehension field. Despite remarkable content reasoning and instruction following capabilities they demonstrated, the hallucination problem of these VideoLLMs is less explored compared with its counterpart in the image domain. To mitigate this gap, we first propose EventHallusion, a novel benchmark that focuses on assessing the VideoLMMs' hallucination phenomenon on video event comprehension. Based on the observation that existing VideoLLMs are entangled with the priors stemming from their foundation models, our EventHallusion is curated by meticulously collecting videos and annotating questions to intentionally mislead the VideoLLMs into interpreting events based on these priors rather than accurately understanding the video content. On the other hand, we also propose a simple yet effective method, called Temporal Contrastive Decoding (TCD), to tackle the hallucination problems of VideoLLMs. The proposed TCD suppresses the model's preference toward their priors by comparing the original video with a constructed counterpart, whose temporal cues are disrupted, during the autoregressive decoding stage. Through comprehensive evaluation of eight open-source and two closed-source VideoLLMs on the proposed EventHallusion benchmark, we find that the open-source models suffer significantly from hallucination problems, whereas the closed-source models perform markedly better. By further equipping open-sourced VideoLLMs with the proposed TCD approach, evident performance improvements are achieved across most metrics in the EventHallusion benchmark. Our codes and benchmark data are available at <a class="link-external link-https" href="https://github.com/Stevetich/EventHallusion" rel="external noopener nofollow">this https URL</a>.

Decompose, Enrich, and Extract! Schema-aware Event Extraction using LLMs

Is a Large Language Model a Good Annotator for Event Extraction?

Mitigating Entity-Level Hallucination in Large Language Models

One Small and One Large for Document-level Event Argument Extraction

ULTRA: Unleash LLMs' Potential for Event Argument Extraction through Hierarchical Modeling and Pair-wise Refinement

PFME: A Modular Approach for Fine-grained Hallucination Detection and Editing of Large Language Models

EventHallusion: Diagnosing Event Hallucinations in Video LLMs

Temporal Insight Enhancement: Mitigating Temporal Hallucination in Multimodal Large Language Models

Detecting and Mitigating the Ungrounded Hallucinations in Text Generation by LLMs

Structured Event Reasoning with Large Language Models

Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigation

Improving Large Language Models in Event Relation Logical Prediction

SLM Meets LLM: Balancing Latency, Interpretability and Consistency in Hallucination Detection

The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models

LMDX: Language Model-based Document Information Extraction and Localization

Event Temporal Relation Extraction based on Retrieval-Augmented on LLMs

Collaborate SLM and LLM with latent answers for event detection

Mitigating Hallucinations in Large Language Models via Self-Refinement-Enhanced Knowledge Retrieval

Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation

Document-Level Event Argument Extraction with Sparse Representation Attention

Harvesting Event Schemas from Large Language Models