Abstract:Vision Language Models (VLMs), which extend Large Language Models (LLM) by incorporating visual understanding capability, have demonstrated significant advancements in addressing open-ended visual question-answering (VQA) tasks. However, these models cannot accurately interpret images infused with text, a common occurrence in real-world scenarios. Standard procedures for extracting information from images often involve learning a fixed set of query embeddings. These embeddings are designed to encapsulate image contexts and are later used as soft prompt inputs in LLMs. Yet, this process is limited to the token count, potentially curtailing the recognition of scenes with text-rich context. To improve upon them, the present study introduces BLIVA: an augmented version of InstructBLIP with Visual Assistant. BLIVA incorporates the query embeddings from InstructBLIP and also directly projects encoded patch embeddings into the LLM, a technique inspired by LLaVA. This approach assists the model to capture intricate details potentially missed during the query decoding process. Empirical evidence demonstrates that our model, BLIVA, significantly enhances performance in processing text-rich VQA benchmarks (up to 17.76% in OCR-VQA benchmark) and in undertaking general (not particularly text-rich) VQA benchmarks (up to 7.9% in Visual Spatial Reasoning benchmark), and achieved 17.72% overall improvement in a comprehensive multimodal LLM benchmark (MME), comparing to our baseline InstructBLIP. BLIVA demonstrates significant capability in decoding real-world images, irrespective of text presence. To demonstrate the broad industry applications enabled by BLIVA, we evaluate the model using a new dataset comprising YouTube thumbnails paired with question-answer sets across 11 diverse categories. Our code and models are freely accessible at <a class="link-external link-https" href="https://github.com/mlpc-ucsd/BLIVA" rel="external noopener nofollow">this https URL</a>.

Enhancing machine vision: the impact of a novel innovative technology on video question-answering

Simple and Effective Visual Question Answering in a Single Modality

VideoQA in the Era of LLMs: An Empirical Study

Right this way: Can VLMs Guide Us to See More to Answer Questions?

QAVidCap: Enhancing Video Captioning Through Question Answering Techniques

ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models

Retrieval-based Video Language Model for Efficient Long Video Question Answering

From Images to Textual Prompts: Zero-Shot Visual Question Answering with Frozen Large Language Models

The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions

LiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering

AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering

Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models

Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey

Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions

Expanding Frozen Vision-Language Models without Retraining: Towards Improved Robot Perception

BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions

Multitask Learning for Visual Question Answering

VQA$^2$:Visual Question Answering for Video Quality Assessment

Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery

LLMs Meet Long Video: Advancing Long Video Question Answering with An Interactive Visual Adapter in LLMs