EAGLE: Towards Efficient Arbitrary Referring Visual Prompts Comprehension for Multimodal Large Language Models

Jiacheng Zhang,Yang Jiao,Shaoxiang Chen,Jingjing Chen,Yu-Gang Jiang

2024-09-26

Abstract:Recently, Multimodal Large Language Models (MLLMs) have sparked great research interests owing to their exceptional content-reasoning and instruction-following capabilities. To effectively instruct an MLLM, in addition to conventional language expressions, the practice of referring to objects by painting with brushes on images has emerged as a prevalent tool (referred to as "referring visual prompts") due to its efficacy in aligning the user's intention with specific image regions. To accommodate the most common referring visual prompts, namely points, boxes, and masks, existing approaches initially utilize specialized feature encoding modules to capture the semantics of the highlighted areas indicated by these prompts. Subsequently, these encoded region features are adapted to MLLMs through fine-tuning on a meticulously curated multimodal instruction dataset. However, such designs suffer from redundancy in architecture. Moreover, they face challenges in effectively generalizing when encountering a diverse range of arbitrary referring visual prompts in real-life scenarios. To address the above issues, we propose EAGLE, a novel MLLM that empowers comprehension of arbitrary referring visual prompts with less training efforts than existing approaches. Specifically, our EAGLE maintains the innate format of the referring visual prompts as colored patches rendered on the given image for conducting the instruction tuning. Our approach embeds referring visual prompts as spatial concepts conveying specific spatial areas comprehensible to the MLLM, with the semantic comprehension of these regions originating from the MLLM itself. Besides, we also propose a Geometry-Agnostic Learning paradigm (GAL) to further disentangle the MLLM's region-level comprehension with the specific formats of referring visual prompts. Extensive experiments are conducted to prove the effectiveness of our proposed method.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the two main challenges faced by existing multimodal large language models (MLLMs) when dealing with referring visual prompts:

1. **Architectural Redundancy**: Existing methods enhance the regional understanding ability of MLLMs by introducing specialized regional feature encoding modules and carefully curated multimodal instruction datasets. However, these designs overlook the regional - level understanding ability that MLLMs already possess, resulting in unnecessary complexity and redundancy.

2. **Poor Generalization Ability**: Existing methods rely on uniformly - formatted referring visual prompts (such as bounding boxes and masks) during training and inference. But in practical applications, users may draw prompts of arbitrary shapes and formats, which are quite different from the prompts in the training data, thus affecting the performance of the model.

To solve these problems, the authors propose EAGLE, a new MLLM, which aims to efficiently understand and process arbitrary referring visual prompts with less training effort. Specifically, EAGLE addresses the above challenges through the following two innovative designs:

- **Maintaining the Original Format of Referring Visual Prompts**: EAGLE renders referring visual prompts of different formats as color patches on the image, as data resources for instruction tuning. This design not only minimizes the modification to existing MLLMs but also makes full use of their inherent regional - level understanding ability.

- **Geometry - Agnostic Learning Paradigm (GAL)**: GAL decouples the relationship between regional - level recognition and prompt geometry by reformulating various referring visual prompts as a set of representative points in a unified format. This enables the MLLM to focus on the mainly referenced objects without being affected by the shape or format of the prompt.

Through these two key designs, EAGLE can show better performance and generalization ability in dealing with arbitrary referring visual prompts than existing methods.