PIN: Positional Insert Unlocks Object Localisation Abilities in VLMs

Michael Dorkenwald,Nimrod Barazani,Cees G. M. Snoek,Yuki M. Asano

2024-02-14

Abstract:Vision-Language Models (VLMs), such as Flamingo and GPT-4V, have shown immense potential by integrating large language models with vision systems. Nevertheless, these models face challenges in the fundamental computer vision task of object localisation, due to their training on multimodal data containing mostly captions without explicit spatial grounding. While it is possible to construct custom, supervised training pipelines with bounding box annotations that integrate with VLMs, these result in specialized and hard-to-scale models. In this paper, we aim to explore the limits of caption-based VLMs and instead propose to tackle the challenge in a simpler manner by i) keeping the weights of a caption-based VLM frozen and ii) not using any supervised detection data. To this end, we introduce an input-agnostic Positional Insert (PIN), a learnable spatial prompt, containing a minimal set of parameters that are slid inside the frozen VLM, unlocking object localisation capabilities. Our PIN module is trained with a simple next-token prediction task on synthetic data without requiring the introduction of new output heads. Our experiments demonstrate strong zero-shot localisation performances on a variety of images, including Pascal VOC, COCO, LVIS, and diverse images like paintings or cartoons.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper mainly discusses how to unlock the object localization ability in title-based Vision-Language Models (VLMs) without additional supervision data or model parameter changes. Existing VLMs face challenges in object localization because they are primarily trained on multimodal data with captions, lacking explicit spatial localization. The paper proposes a learning-based spatial cue called "Position Insertion" (PIN), which is a lightweight parameter vector that can be slid in a frozen VLM to enhance the model's spatial awareness. The PIN module is trained on synthetic data through a simple next token prediction task, which includes rendered synthetic objects and background images, providing precise ground truth locations. Experimental results demonstrate the strong zero-shot localization performance of PIN on various image datasets such as Pascal VOC, COCO, and LVIS. The paper also compares PIN with other methods, proving that PIN effectively enhances the object localization capability of VLM without sacrificing its general ability.