Abstract:Vision language models (VLMs) have seen growing adoption in recent years, but many still struggle with basic spatial reasoning errors. We hypothesize that this is due to VLMs adopting pre-trained vision backbones, specifically vision transformers (ViTs) trained with image-level supervision and minimal inductive biases. Such models may fail to encode the class contents at each position in the image, and our goal is to resolve this by ensuring that the vision backbone effectively captures both local and global image semantics. Our main insight is that we do not require new supervision to learn this capability -- pre-trained models contain significant knowledge of local semantics that we can extract and use for scalable self-supervision. We propose a new efficient post-training stage for ViTs called locality alignment and a novel fine-tuning procedure called MaskEmbed that uses a masked reconstruction loss to learn semantic contributions for each image patch. We first evaluate locality alignment with a vision-only benchmark, finding that it improves a model's performance at a patch-level semantic segmentation task, especially for strong backbones trained with image-caption pairs (e.g., CLIP and SigLIP). We then train a series of VLMs with and without locality alignment, and show that locality-aligned backbones improve performance across a range of benchmarks, particularly ones that involve spatial understanding (e.g., RefCOCO, OCID-Ref, TallyQA, VSR, AI2D). Overall, we demonstrate that we can efficiently learn local semantic extraction via a locality alignment stage, and that this procedure complements existing VLM training recipes that use off-the-shelf vision backbones.

TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification

Tag2Text: Guiding Vision-Language Model via Image Tagging

HybridVocab: Towards Multi-Modal Machine Translation Via Multi-Aspect Alignment

Noisy Tag Alignment with Image Regions.

Semantic Aligned Multi-modal Transformer for Vision-LanguageUnderstanding: A Preliminary Study on Visual QA

Enhancing the Alignment Between Target Words and Corresponding Frames for Video Captioning.

Contrastive Vision-Language Alignment Makes Efficient Instruction Learner

Align and Tell: Boosting Text-Video Retrieval With Local Alignment and Fine-Grained Supervision

AGREE: Aligning Cross-Modal Entities for Image-Text Retrieval Upon Vision-Language Pre-trained Models

Unified Lexical Representation for Interpretable Visual-Language Alignment

Fine-Granularity Alignment for Text-Based Person Retrieval Via Semantics-Centric Visual Division

Learning Video-Text Aligned Representations for Video Captioning

Multi-Grained Cross-modal Alignment for Learning Open-vocabulary Semantic Segmentation from Text Supervision

Locality Alignment Improves Vision-Language Models

Multimodal Features Alignment for Vision–Language Object Tracking

Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations

Assessing and Learning Alignment of Unimodal Vision and Language Models

Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models

TagCLIP: Improving Discrimination Ability of Open-Vocabulary Semantic Segmentation

Removing Distributional Discrepancies in Captions Improves Image-Text Alignment

Text-Vision Relationship Alignment for Referring Image Segmentation