Abstract:Vision Language Models (VLMs), which extend Large Language Models (LLM) by incorporating visual understanding capability, have demonstrated significant advancements in addressing open-ended visual question-answering (VQA) tasks. However, these models cannot accurately interpret images infused with text, a common occurrence in real-world scenarios. Standard procedures for extracting information from images often involve learning a fixed set of query embeddings. These embeddings are designed to encapsulate image contexts and are later used as soft prompt inputs in LLMs. Yet, this process is limited to the token count, potentially curtailing the recognition of scenes with text-rich context. To improve upon them, the present study introduces BLIVA: an augmented version of InstructBLIP with Visual Assistant. BLIVA incorporates the query embeddings from InstructBLIP and also directly projects encoded patch embeddings into the LLM, a technique inspired by LLaVA. This approach assists the model to capture intricate details potentially missed during the query decoding process. Empirical evidence demonstrates that our model, BLIVA, significantly enhances performance in processing text-rich VQA benchmarks (up to 17.76% in OCR-VQA benchmark) and in undertaking general (not particularly text-rich) VQA benchmarks (up to 7.9% in Visual Spatial Reasoning benchmark), and achieved 17.72% overall improvement in a comprehensive multimodal LLM benchmark (MME), comparing to our baseline InstructBLIP. BLIVA demonstrates significant capability in decoding real-world images, irrespective of text presence. To demonstrate the broad industry applications enabled by BLIVA, we evaluate the model using a new dataset comprising YouTube thumbnails paired with question-answer sets across 11 diverse categories. For researchers interested in further exploration, our code and models are freely accessible at https://github.com/mlpc-ucsd/BLIVA.

Black-box Model Ensembling for Textual and Visual Question Answering via Information Fusion

Simple and Effective Visual Question Answering in a Single Modality

Information Fusion in Visual Question Answering: A Survey

Enhancing Visual Question Answering through Ranking-Based Hybrid Training and Multimodal Fusion

Context-aware Multi-level Question Embedding Fusion for visual question answering

Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering

Selectively Answering Visual Questions

Two-Stage Multimodality Fusion for High-Performance Text-Based Visual Question Answering.

Greedy Gradient Ensemble for Robust Visual Question Answering

Parallel Fusion of Graph and Text with Semantic Enhancement for Commonsense Question Answering

Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

Collaborative Modality Fusion for Mitigating Language Bias in Visual Question Answering

Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models

Self-Bootstrapped Visual-Language Model for Knowledge Selection and Question Answering

LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion

BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions

Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models

Multitask Learning for Visual Question Answering

Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering.

How to find a good image-text embedding for remote sensing visual question answering?