Abstract:Benefiting from the generalization capability of CLIP, recent vision language pre-training (VLP) models have demonstrated an impressive ability to capture virtually any visual concept in daily images. However, due to the presence of unseen categories in open-vocabulary settings, existing algorithms struggle to effectively capture strong semantic correlations between categories, resulting in sub-optimal performance on the open-vocabulary multi-label recognition (OV-MLR). Furthermore, the substantial variation in the number of discriminative areas across diverse object categories is misaligned with the fixed-number patch matching used in current methods, introducing noisy visual cues that hinder the accurate capture of target semantics. To tackle these challenges, we propose a novel category-adaptive cross-modal semantic refinement and transfer (C$^2$SRT) framework to explore the semantic correlation both within each category and across different categories, in a category-adaptive manner. The proposed framework consists of two complementary modules, i.e., intra-category semantic refinement (ISR) module and inter-category semantic transfer (IST) module. Specifically, the ISR module leverages the cross-modal knowledge of the VLP model to adaptively find a set of local discriminative regions that best represent the semantics of the target category. The IST module adaptively discovers a set of most correlated categories for a target category by utilizing the commonsense capabilities of LLMs to construct a category-adaptive correlation graph and transfers semantic knowledge from the correlated seen categories to unseen ones. Extensive experiments on OV-MLR benchmarks clearly demonstrate that the proposed C$^2$SRT framework outperforms current state-of-the-art algorithms.

A Refinement Framework for Cross Language Text Categorization.

Cross Language Text Categorization Using a Bilingual Lexicon.

A multiclass classification framework for document categorization

Aggressive Dimensionality Reduction With Reinforcement Local Feature Selection For Text Categorization

Refining Image Categorization by Exploiting Web Images and General Corpus.

Language Independent Text Categorization.

Cross-lingual Dataless Classification for Languages with Small Wikipedia Presence

Statistical Analysis for Chinese-English Verb Subcategorization

Exploiting Textual and Visual Features for Image Categorization

Category-Adaptive Cross-Modal Semantic Refinement and Transfer for Open-Vocabulary Multi-Label Recognition

Chinese Document Categorization without Dictionary Support and Segmentation Processing

Bilingual query translation and expansion for supporting more effective cross-language image retrieval.

Improving Text Categorization Using the Importance of Words in Different Categories

Chinese-English Cross-Lingual Information Retrieval Based On Domain Ontology Knowledge

Building a bilingual WordNet-like lexicon: the new approach and algorithms

Automatic Generation of Language-Independent Features for Cross-Lingual Classification

Learning Inter-Related Statistical Query Translation Models for English-Chinese Bi-Directional CLIR

Universal Cross-Lingual Text Classification

Re-ranking for Bilingual Lexicon Extraction with Bi-directional Linear Transformation from Comparable Corpora.

Effective Bilingual Constraints for Semi-Supervised Learning of Named Entity Recognizers.

A Hybrid Chinese Language Model based on a Combination of Ontology with Statistical Method