Abstract:Vision Language Models (VLMs), which extend Large Language Models (LLM) by incorporating visual understanding capability, have demonstrated significant advancements in addressing open-ended visual question-answering (VQA) tasks. However, these models cannot accurately interpret images infused with text, a common occurrence in real-world scenarios. Standard procedures for extracting information from images often involve learning a fixed set of query embeddings. These embeddings are designed to encapsulate image contexts and are later used as soft prompt inputs in LLMs. Yet, this process is limited to the token count, potentially curtailing the recognition of scenes with text-rich context. To improve upon them, the present study introduces BLIVA: an augmented version of InstructBLIP with Visual Assistant. BLIVA incorporates the query embeddings from InstructBLIP and also directly projects encoded patch embeddings into the LLM, a technique inspired by LLaVA. This approach assists the model to capture intricate details potentially missed during the query decoding process. Empirical evidence demonstrates that our model, BLIVA, significantly enhances performance in processing text-rich VQA benchmarks (up to 17.76% in OCR-VQA benchmark) and in undertaking general (not particularly text-rich) VQA benchmarks (up to 7.9% in Visual Spatial Reasoning benchmark), and achieved 17.72% overall improvement in a comprehensive multimodal LLM benchmark (MME), comparing to our baseline InstructBLIP. BLIVA demonstrates significant capability in decoding real-world images, irrespective of text presence. To demonstrate the broad industry applications enabled by BLIVA, we evaluate the model using a new dataset comprising YouTube thumbnails paired with question-answer sets across 11 diverse categories. Our code and models are freely accessible at <a class="link-external link-https" href="https://github.com/mlpc-ucsd/BLIVA" rel="external noopener nofollow">this https URL</a>.

Visual-audio correspondence and its effect on video tipping: Evidence from Bilibili vlogs

The secret of voice: How acoustic characteristics affect video creators' performance on Bilibili

How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition

Crowding-Out in Content Monetization under Pay What You Want: Evidence from Live Streaming

In-Consumption Information Cues and Online Video Consumption

From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation

Tell What You Hear From What You See -- Video to Audio Generation Through Text

The Bright Side of Emotional Extremity: Evidence from Tipping in Live Streaming Platform

Themes Informed Audio-visual Correspondence Learning

Perceptual Evaluation of 360 Audiovisual Quality and Machine Learning Predictions

Predicting video popularity based on video covers and titles using a multimodal large-scale model and pipeline parallelism

UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content

Forecasting the broadcast volume of cultural communication assisted by short video technology and analyzing the influencing factors

BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions

Visual Quality Assessment for Web Videos

Exploring user engagement behavior with short-form video advertising on short-form video platforms: a visual-audio perspective

Telepresence Video Quality Assessment

What You See Is What Matters: A Novel Visual and Physics-Based Metric for Evaluating Video Generation Quality

Enhancing Micro-video Understanding by Harnessing External Sounds

PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data