Abstract:Vision Language Models (VLMs), which extend Large Language Models (LLM) by incorporating visual understanding capability, have demonstrated significant advancements in addressing open-ended visual question-answering (VQA) tasks. However, these models cannot accurately interpret images infused with text, a common occurrence in real-world scenarios. Standard procedures for extracting information from images often involve learning a fixed set of query embeddings. These embeddings are designed to encapsulate image contexts and are later used as soft prompt inputs in LLMs. Yet, this process is limited to the token count, potentially curtailing the recognition of scenes with text-rich context. To improve upon them, the present study introduces BLIVA: an augmented version of InstructBLIP with Visual Assistant. BLIVA incorporates the query embeddings from InstructBLIP and also directly projects encoded patch embeddings into the LLM, a technique inspired by LLaVA. This approach assists the model to capture intricate details potentially missed during the query decoding process. Empirical evidence demonstrates that our model, BLIVA, significantly enhances performance in processing text-rich VQA benchmarks (up to 17.76% in OCR-VQA benchmark) and in undertaking general (not particularly text-rich) VQA benchmarks (up to 7.9% in Visual Spatial Reasoning benchmark), and achieved 17.72% overall improvement in a comprehensive multimodal LLM benchmark (MME), comparing to our baseline InstructBLIP. BLIVA demonstrates significant capability in decoding real-world images, irrespective of text presence. To demonstrate the broad industry applications enabled by BLIVA, we evaluate the model using a new dataset comprising YouTube thumbnails paired with question-answer sets across 11 diverse categories. For researchers interested in further exploration, our code and models are freely accessible at https://github.com/mlpc-ucsd/BLIVA.

Infrared-LLaVA: Enhancing Understanding of Infrared Images in Multi-Modal Large Language Models

INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Model

MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning

BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions

TC-LLaVA: Rethinking the Transfer from Image to Video Understanding with Temporal Considerations

SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant

u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model

InfMLLM: A Unified Framework for Visual-Language Tasks.

OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding

LLaVA-KD: A Framework of Distilling Multimodal Large Language Models

Infrared Image Captioning Based on Unsupervised Learning and Reinforcement Learning

LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models

Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey

Visually-Augmented Language Modeling

Infrared Image Generation Based on Visual State Space and Contrastive Learning

Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models

LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture

X-VILA: Cross-Modality Alignment for Large Language Model

Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models