LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Description

Yizhang Jin,Jian Li,Jiangning Zhang,Jianlong Hu,Zhenye Gan,Xin Tan,Yong Liu,Yabiao Wang,Chengjie Wang,Lizhuang Ma

2024-10-30

Abstract:Visual Spatial Description (VSD) aims to generate texts that describe the spatial relationships between objects within images. Traditional visual spatial relationship classification (VSRC) methods typically output the spatial relationship between two objects in an image, often neglecting world knowledge and lacking general language capabilities. In this paper, we propose a Large Language-and-Vision Assistant for Visual Spatial Description, named LLaVA-VSD, which is designed for the classification, description, and open-ended description of visual spatial relationships. Specifically, the model first constructs a VSD instruction-following dataset using given figure-caption pairs for the three tasks. It then employs LoRA to fine-tune a Large Language and Vision Assistant for VSD, which has 13 billion parameters and supports high-resolution images. Finally, a large language model (Qwen-2) is used to refine the generated sentences, enhancing their diversity and accuracy. LLaVA-VSD demonstrates excellent multimodal conversational capabilities and can follow open-ended instructions to assist with inquiries about object relationships in images.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The paper attempts to address two major limitations in existing Visual Spatial Description (VSD) tasks: the lack of sufficient prior knowledge and suboptimal text generation capabilities.

Specifically:

1. **Lack of Prior Knowledge**: Existing VSD models (such as VL-BART and VL-T5) are typically trained on smaller, task-specific datasets, thus lacking extensive world knowledge. This limits their ability to generate accurate and context-rich spatial descriptions.

2. **Suboptimal Text Generation Capabilities**: Current models perform poorly in generating high-quality, context-appropriate detailed spatial descriptions, especially in following complex instructions, resulting in descriptions that are too generic or fail to capture specific contextual details.

To address these issues, the paper proposes a new method called LLaVA-VSD, which leverages Multimodal Large Language Models (MLLMs) to enhance the VSD task. By converting the VSD dataset into a visual instruction format, fine-tuning the LLaVA model using Low-Rank Adaptation (LoRA), and optimizing the generated descriptions through a pure text large language model, this method significantly improves the accuracy, contextual richness, and diversity of spatial descriptions.