Abstract:While vision-language models like CLIP have shown remarkable success in open-vocabulary tasks, their application is currently confined to image-level tasks, and they still struggle with dense predictions. Recent works often attribute such deficiency in dense predictions to the self-attention layers in the final block, and have achieved commendable results by modifying the original query-key attention to self-correlation attention, (e.g., query-query and key-key attention). However, these methods overlook the cross-correlation attention (query-key) properties, which capture the rich spatial correspondence. In this paper, we reveal that the cross-correlation of the self-attention in CLIP's non-final layers also exhibits localization properties. Therefore, we propose the Residual Cross-correlation Self-attention (RCS) module, which leverages the cross-correlation self-attention from intermediate layers to remold the attention in the final block. The RCS module effectively reorganizes spatial information, unleashing the localization potential within CLIP for dense vision-language inference. Furthermore, to enhance the focus on regions of the same categories and local consistency, we propose the Semantic Feedback Refinement (SFR) module, which utilizes semantic segmentation maps to further adjust the attention scores. By integrating these two strategies, our method, termed ResCLIP, can be easily incorporated into existing approaches as a plug-and-play module, significantly boosting their performance in dense vision-language inference. Extensive experiments across multiple standard benchmarks demonstrate that our method surpasses state-of-the-art training-free methods, validating the effectiveness of the proposed approach. Code is available at <a class="link-external link-https" href="https://github.com/yvhangyang/ResCLIP" rel="external noopener nofollow">this https URL</a>.

Adapting CLIP For Phrase Localization Without Further Training

Contrastive Localized Language-Image Pre-Training

CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding

Can CLIP Help Sound Source Localization?

CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction

Phrase Localization Without Paired Training Examples

Phrase-level Prediction for Video Temporal Localization

SpaceCLIP: A Vision-Language Pretraining Framework With Spatial Reconstruction On Text

Improving CLIP Training with Language Rewrites

How Much Can CLIP Benefit Vision-and-Language Tasks?

Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentation

ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation

CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement

ResCLIP: Residual Attention for Training-free Dense Vision-language Inference

Fine-tuning CLIP Text Encoders with Two-step Paraphrasing

Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentation

TagCLIP: A Local-to-Global Framework to Enhance Open-Vocabulary Multi-Label Classification of CLIP Without Training

Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cues

FiGCLIP: Fine-Grained CLIP Adaptation via Densely Annotated Videos

LatteCLIP: Unsupervised CLIP Fine-Tuning via LMM-Synthetic Texts