Abstract:Vision Language Models (VLMs), which extend Large Language Models (LLM) by incorporating visual understanding capability, have demonstrated significant advancements in addressing open-ended visual question-answering (VQA) tasks. However, these models cannot accurately interpret images infused with text, a common occurrence in real-world scenarios. Standard procedures for extracting information from images often involve learning a fixed set of query embeddings. These embeddings are designed to encapsulate image contexts and are later used as soft prompt inputs in LLMs. Yet, this process is limited to the token count, potentially curtailing the recognition of scenes with text-rich context. To improve upon them, the present study introduces BLIVA: an augmented version of InstructBLIP with Visual Assistant. BLIVA incorporates the query embeddings from InstructBLIP and also directly projects encoded patch embeddings into the LLM, a technique inspired by LLaVA. This approach assists the model to capture intricate details potentially missed during the query decoding process. Empirical evidence demonstrates that our model, BLIVA, significantly enhances performance in processing text-rich VQA benchmarks (up to 17.76% in OCR-VQA benchmark) and in undertaking general (not particularly text-rich) VQA benchmarks (up to 7.9% in Visual Spatial Reasoning benchmark), and achieved 17.72% overall improvement in a comprehensive multimodal LLM benchmark (MME), comparing to our baseline InstructBLIP. BLIVA demonstrates significant capability in decoding real-world images, irrespective of text presence. To demonstrate the broad industry applications enabled by BLIVA, we evaluate the model using a new dataset comprising YouTube thumbnails paired with question-answer sets across 11 diverse categories. Our code and models are freely accessible at <a class="link-external link-https" href="https://github.com/mlpc-ucsd/BLIVA" rel="external noopener nofollow">this https URL</a>.

LiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering

A Simple LLM Framework for Long-Range Video Question-Answering

VideoQA in the Era of LLMs: An Empirical Study

Retrieval-based Video Language Model for Efficient Long Video Question Answering

Language-Guided Visual Aggregation Network for Video Question Answering

BoViLA: Bootstrapping Video-Language Alignment via LLM-Based Self-Questioning and Answering

Video Question Answering with Semantic Disentanglement and Reasoning

Prompting Large Language Models with Fine-Grained Visual Relations from Scene Graph for Visual Question Answering

ViLA: Efficient Video-Language Alignment for Video Question Answering

Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models

LLMs Meet Long Video: Advancing Long Video Question Answering with An Interactive Visual Adapter in LLMs

VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection

VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding

VideoDistill: Language-aware Vision Distillation for Video Question Answering

BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions

Language-aware Visual Semantic Distillation for Video Question Answering

Enhancing machine vision: the impact of a novel innovative technology on video question-answering

Question-Instructed Visual Descriptions for Zero-Shot Video Question Answering

LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling