Abstract:Multi-label image recognition aims to predict a set of labels that present in an image. The key to deal with such problem is to mine the associations between image contents and labels, and further obtain the correct assignments between images and their labels. In this paper, we treat each image as a bag of instances, and formulate the task of multi-label image recognition as an instance-label matching selection problem. To model such problem, we propose an innovative Semantic-aware Graph Matching framework for Multi-Label image recognition (ML-SGM), in which Graph Matching mechanism is introduced owing to its good performance of excavating the instance and label relationship. The framework explicitly establishes category correlations and instance-label correspondences by modeling the relation among content-aware (instance) and semantic-aware (label) category representations, to facilitate multi-label image understanding and reduce the dependency of large amounts of training samples for each category. Specifically, we first construct an instance spatial graph and a label semantic graph respectively and then incorporate them into a constructed assignment graph by connecting each instance to all labels. Subsequently, the graph network block is adopted to aggregate and update all nodes and edges state on the assignment graph to form structured representations for each instance and label. Our network finally derives a prediction score for each instance-label correspondence and optimizes such correspondence with a weighted cross-entropy loss. Empirical results conducted on generic multi-label image recognition demonstrate the superiority of our proposed method. Moreover, the proposed method also shows advantages in multi-label recognition with partial labels and multi-label few-shot learning, as well as outperforms current state-of-the-art methods with a clear margin. Our code is available at https://github.com/yananwu0510/ML-SGM.

Exploring Structured Semantic Prior for Multi Label Recognition with Incomplete Labels

Learning Semantic-Aware Representation in Visual-Language Models for Multi-Label Recognition with Partial Labels

Hierarchical Prompt Learning Using CLIP for Multi-label Classification with Single Positive Labels

Multi-Level Label Correction by Distilling Proximate Patterns for Semi-supervised Semantic Segmentation

Semi-Supervised Dual Relation Learning for Multi-Label Classification

Semantic-Guided Representation Enhancement for Multi-Label Image Classification

Semantic Representation and Dependency Learning for Multi-Label Image Recognition

Incomplete Multi-View Partial Multi-Label Classification Via Deep Semantic Structure Preservation

Semantic Supplementary Network With Prior Information for Multi-Label Image Classification

Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition

Improving Multi-label Learning with Missing Labels by Structured Semantic Correlations

Context-Based Semantic-Aware Alignment for Semi-Supervised Multi-Label Learning

Semantic Role Labeling Integrated with Multilevel Linguistic Cues and Bi-LSTM-CRF

Deep Learning for Multilabel Remote Sensing Image Annotation with Dual-Level Semantic Concepts

A Multi-Level Label-Aware Semi-Supervised Framework for Remote Sensing Scene Classification.

Semantic-Aware Graph Matching Mechanism for Multi-Label Image Recognition

Heterogeneous Semantic Transfer for Multi-label Recognition with Partial Labels

Structured Semantic Transfer for Multi-Label Recognition with Partial Labels

Multi-layered Semantic Representation Network for Multi-label Image Classification

Multi-label learning with semantic embeddings

Correlation-Induced Label Prior for Semi-Supervised Multi-Label Learning