Abstract:Despite extensive research, open-vocabulary segmentation methods still struggle to generalize across diverse domains. To reduce the computational cost of adapting Vision-Language Models (VLMs) while preserving their pre-trained knowledge, most methods freeze the VLMs for mask classification and train only the mask generator. However, our comprehensive analysis reveals a surprising insight: open-vocabulary segmentation is primarily bottlenecked by mask classification, not mask generation. This discovery prompts us to rethink the existing paradigm and explore an alternative approach. Instead of freezing the VLM, we propose to freeze the pre-trained mask generator and focus on optimizing the mask classifier. Building on the observation that VLMs pre-trained on global-pooled image-text features often fail to capture fine-grained semantics necessary for effective mask classification, we propose a novel Fine-grained Semantic Adaptation (FISA) method to address this limitation. FISA enhances the extracted visual features with fine-grained semantic awareness by explicitly integrating this crucial semantic information early in the visual encoding process. As our method strategically optimizes only a small portion of the VLM's parameters, it enjoys the efficiency of adapting to new data distributions while largely preserving the valuable VLM pre-trained knowledge. Extensive ablation studies confirm the superiority of our approach. Notably, FISA achieves new state-of-the-art results across multiple representative benchmarks, improving performance by up to +1.0 PQ and +3.0 mIoU and reduces training costs by nearly 5x compared to previous best methods. Our code and data will be made public.

Improved Cluster Center Adaption for Image Classification

Improving VLAD with Regional PCA Whitening.

Image Representation Optimization Based on Locally Aggregated Descriptors.

Codebook Enhancement of Vlad Representation for Visual Recognition.

Fine-residual VLAD for Image Retrieval.

Visual Object Recognition Method Based on Feature Vocabulary Tree and Energy Minimization

A Simple but Efficient Way to Combine Vlad with Locality-Constrained Linear Coding

Incorporating Visual Adjectives for Image Classification

An Improved Averaging Combination Method for Image and Object Recognition

Hierarchical Multi-Vlad For Image Retrieval

Boosting VLAD with Supervised Dictionary Learning and High-Order Statistics.

Object Recognition via Adaptive Multi-level Feature Integration

Visual Vocabulary Optimization with Spatial Context for Image Annotation and Classification

Dual Cluster Contrastive learning for Object Re-Identification

Improved Soft Assignment Coding for Image Classification

Visual Recognition Using Density Adaptive Clustering

Adapting Vision-Language Model with Fine-grained Semantics for Open-Vocabulary Segmentation

Exploiting Feature Correspondence Constraints for Image Recognition

Novel fuzzy clustering algorithm with variable multi-pixel fitting spatial information for image segmentation

Image Matching Optimization Based on Taguchi Method and Adaptive Spatial Clustering with SIFT Features.

Center-adaptive weighted binary K-means for image clustering