Abstract:Vision Language Models (VLMs), which extend Large Language Models (LLM) by incorporating visual understanding capability, have demonstrated significant advancements in addressing open-ended visual question-answering (VQA) tasks. However, these models cannot accurately interpret images infused with text, a common occurrence in real-world scenarios. Standard procedures for extracting information from images often involve learning a fixed set of query embeddings. These embeddings are designed to encapsulate image contexts and are later used as soft prompt inputs in LLMs. Yet, this process is limited to the token count, potentially curtailing the recognition of scenes with text-rich context. To improve upon them, the present study introduces BLIVA: an augmented version of InstructBLIP with Visual Assistant. BLIVA incorporates the query embeddings from InstructBLIP and also directly projects encoded patch embeddings into the LLM, a technique inspired by LLaVA. This approach assists the model to capture intricate details potentially missed during the query decoding process. Empirical evidence demonstrates that our model, BLIVA, significantly enhances performance in processing text-rich VQA benchmarks (up to 17.76% in OCR-VQA benchmark) and in undertaking general (not particularly text-rich) VQA benchmarks (up to 7.9% in Visual Spatial Reasoning benchmark), and achieved 17.72% overall improvement in a comprehensive multimodal LLM benchmark (MME), comparing to our baseline InstructBLIP. BLIVA demonstrates significant capability in decoding real-world images, irrespective of text presence. To demonstrate the broad industry applications enabled by BLIVA, we evaluate the model using a new dataset comprising YouTube thumbnails paired with question-answer sets across 11 diverse categories. For researchers interested in further exploration, our code and models are freely accessible at https://github.com/mlpc-ucsd/BLIVA.

From Images to Textual Prompts: Zero-Shot Visual Question Answering with Frozen Large Language Models

Simple and Effective Visual Question Answering in a Single Modality

Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Prompts

ZVQAF: Zero-shot visual question answering with feedback from large language models

Few-Shot VQA with Frozen LLMs: A Tale of Two Approaches

Good Questions Help Zero-Shot Image Reasoning

Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions

Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training

Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models

Zero-shot Visual Question Answering with Language Model Feedback

Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts

BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions

Do LLMs Understand Visual Anomalies? Uncovering LLM's Capabilities in Zero-shot Anomaly Detection

Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering

Prompting Large Language Models with Fine-Grained Visual Relations from Scene Graph for Visual Question Answering

Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering

InfMLLM: A Unified Framework for Visual-Language Tasks.

Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks

Right this way: Can VLMs Guide Us to See More to Answer Questions?

Enhancing machine vision: the impact of a novel innovative technology on video question-answering