Abstract:Multi-label image recognition aims to predict a set of labels that present in an image. The key to deal with such problem is to mine the associations between image contents and labels, and further obtain the correct assignments between images and their labels. In this paper, we treat each image as a bag of instances, and formulate the task of multi-label image recognition as an instance-label matching selection problem. To model such problem, we propose an innovative Semantic-aware Graph Matching framework for Multi-Label image recognition (ML-SGM), in which Graph Matching mechanism is introduced owing to its good performance of excavating the instance and label relationship. The framework explicitly establishes category correlations and instance-label correspondences by modeling the relation among content-aware (instance) and semantic-aware (label) category representations, to facilitate multi-label image understanding and reduce the dependency of large amounts of training samples for each category. Specifically, we first construct an instance spatial graph and a label semantic graph respectively and then incorporate them into a constructed assignment graph by connecting each instance to all labels. Subsequently, the graph network block is adopted to aggregate and update all nodes and edges state on the assignment graph to form structured representations for each instance and label. Our network finally derives a prediction score for each instance-label correspondence and optimizes such correspondence with a weighted cross-entropy loss. Empirical results conducted on generic multi-label image recognition demonstrate the superiority of our proposed method. Moreover, the proposed method also shows advantages in multi-label recognition with partial labels and multi-label few-shot learning, as well as outperforms current state-of-the-art methods with a clear margin. Our code is available at https://github.com/yananwu0510/ML-SGM.

Syntactic-Informed Graph Networks for Sentence Matching

Visual-Semantic Graph Matching for Visual Grounding

Graph Structured Network for Image-Text Matching

Learning Better Sentence Representation with Syntax Information

RI-Match: Integrating Both Representations and Interactions for Deep Semantic Matching.

Stochastic Iterative Graph Matching

Scene graph semantic inference for image and text matching

Joint Graph Learning and Matching for Semantic Feature Correspondence

Exploring fine-grained syntactic information for aspect-based sentiment classification with dual graph neural networks

SEGMN: A Structure-Enhanced Graph Matching Network for Graph Similarity Learning

A Short-Text Similarity Model Combining Semantic and Syntactic Information

Shortcut Enhanced Syntactic and Semantic Dual-channel Network for Aspect-based Sentiment Analysis

Local and Global: Text Matching Via Syntax Graph Calibration

Multiresolution Graph Attention Networks for Relevance Matching

Status-Aware Signed Heterogeneous Network Embedding With Graph Neural Networks

A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations

Cross-modal Graph Matching Network for Image-text Retrieval

An Unsupervised Sampling Approach for Image-Sentence Matching Using Document-Level Structural Information

Multi-Sentence Matching via Exploiting List-level Semantics Expansion

Similarity-based Neighbor Selection for Graph LLMs

Semantic-Aware Graph Matching Mechanism for Multi-Label Image Recognition