Abstract:The remarkable advancements in Multimodal Large Language Models (MLLMs) have not rendered them immune to challenges, particularly in the context of handling deceptive information in prompts, thus producing hallucinated responses under such conditions. To quantitatively assess this vulnerability, we present MAD-Bench, a carefully curated benchmark that contains 1000 test samples divided into 5 categories, such as non-existent objects, count of objects, and spatial relationship. We provide a comprehensive analysis of popular MLLMs, ranging from GPT-4v, Reka, Gemini-Pro, to open-sourced models, such as LLaVA-NeXT and MiniCPM-Llama3. Empirically, we observe significant performance gaps between GPT-4o and other models; and previous robust instruction-tuned models are not effective on this new benchmark. While GPT-4o achieves 82.82% accuracy on MAD-Bench, the accuracy of any other model in our experiments ranges from 9% to 50%. We further propose a remedy that adds an additional paragraph to the deceptive prompts to encourage models to think twice before answering the question. Surprisingly, this simple method can even double the accuracy; however, the absolute numbers are still too low to be satisfactory. We hope MAD-Bench can serve as a valuable benchmark to stimulate further research to enhance model resilience against deceptive prompts.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that multimodal large language models (MLLMs) are prone to hallucination, that is, generating responses that do not match the actual situation, when faced with deceptive prompts. Specifically, the authors constructed a new benchmark test set, MAD - Bench, which contains 1,000 image - prompt pairs, divided into five different deception categories, to systematically evaluate the ability of MLLMs to handle inconsistent text prompts and images. Through this benchmark test, the researchers hope to reveal the current vulnerability of MLLMs in this regard and propose a simple improvement method to improve the robustness of the model. ### Main Findings: 1. **Differences in Model Performance**: GPT - 4V performs significantly better than other models on MAD - Bench, but even GPT - 4V has significant room for improvement. 2. **Common Error Types**: - **Inaccurate Object Detection**: The model may misidentify non - existent objects. - **Redundant Object Identification**: The model may mistake one object for multiple objects. - **Speculating Invisible Objects**: The model may fabricate descriptions of non - existent objects based on its internal knowledge base. - **Inconsistent Reasoning**: The model may waver between following the prompt information and the actual image content when generating responses. ### Improvement Method: The authors proposed a simple method by adding an extra piece of text to the prompt, encouraging the model to think carefully before answering the question. This method was tested on multiple models, and the results showed that it can significantly improve the accuracy of the model, although the absolute value still needs to be improved. ### Future Directions: 1. **Training Data**: Create a subset of training data containing deceptive prompts and train MLLMs to resist deception. 2. **Checking the Consistency between Images and Prompts**: Identify and interpret elements in the image, analyze the content and intention of the question, and compare the differences between the two. 3. **Focusing on Factual Information**: Ensure that responses are based only on the factual information in the image and avoid speculation beyond the scope of the image and the question. Through these studies, the authors hope to inspire more research to enhance the robustness of MLLMs when faced with deceptive prompts.

How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts

A Debate-Driven Experiment on LLM Hallucinations and Accuracy

LLM Whisperer: An Inconspicuous Attack to Bias LLM Responses

An LLM can Fool Itself: A Prompt-Based Adversarial Attack

Large Language Models as Misleading Assistants in Conversation

An Assessment of Model-On-Model Deception

Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly

Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?

LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples

Self-Deception: Reverse Penetrating the Semantic Firewall of Large Language Models

Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method

Why Are My Prompts Leaked? Unraveling Prompt Extraction Threats in Customized Large Language Models

Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information

Refusing Safe Prompts for Multi-modal Large Language Models

Do Large Language Models Exhibit Cognitive Dissonance? Studying the Difference Between Revealed Beliefs and Stated Answers

Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios

How Susceptible are LLMs to Influence in Prompts?

Are You Human? An Adversarial Benchmark to Expose LLMs

Difficult Task Yes but Simple Task No: Unveiling the Laziness in Multimodal LLMs

DecoPrompt : Decoding Prompts Reduces Hallucinations when Large Language Models Meet False Premises

Prompting with Divide-and-Conquer Program Makes Large Language Models Discerning to Hallucination and Deception