Abstract:Multi-label image classification is an essential yet challenging task that requires to recognize multiple objects of images. To this end, recent studies have sought to acquire visual representations for each label by attention models, and then train binary classifiers for prediction. However, these methods have two major drawbacks: 1) They rely heavily on the precise alignments between two modalities, which is still challenging for current attention models; 2) They ignore patch-level representations rich in local object features, which are also of great importance for label recognition. In this paper, we propose a semantic-guided representation enhancement framework, which augments patch-level representations with object-level representations for robust label recognition. Concretely, the proposed framework consists of two significant components: 1) an inter-modal attention module that accounts for coarsely locating object regions and producing object-level representations for each label; 2) an intra-modal attention module that aggregates object representations to enhance patch representations based on their correlations. In this way, both local clues and global glances of objects are fully exploited simultaneously, rather than relying solely on object-level representations obtained by the inter-modal attention, thus improving the performance of label recognition. Experimental results show that our framework outperforms the state-of-the-art methods by 0.5%, 0.6%, 0.7% and 0.8% in mAP on Pascal VOC 2007, Microsoft COCO, NUS-WIDE and Visual Genome datasets, respectively. Codes and models are available on https://github.com/jasonseu/SGRE.

Joint recognition / segmentation with cascaded multi-level feature classification and confidence propagation

Improving Semantic Image Segmentation with a Probabilistic Superpixel-Based Dense Conditional Random Field.

Semantic Segmentation Via Structured Refined Prediction and Dual Global Priors.

Multi-class Object Segmentation Based on Jointly Integrating Segment-Level and Image-Level Object Priors

Superpixel-Based Long-Range Dependent Network for High-Resolution Remote-Sensing Image Classification

Semantic Segmentation Via Structured Patch Prediction, Context Crf And Guidance Crf

Multi-Scale Semantic Model for Unsupervised Object Segmentation

Instance-aware Semantic Segmentation via Multi-task Network Cascades

Deep Object Co-segmentation via Spatial-Semantic Network Modulation

Semantic-Guided Representation Enhancement for Multi-Label Image Classification

Semantic Image Segmentation Via Guidance of Image Classification

Superpixel-based Object Class Segmentation Using Conditional Random Fields.

Object Proposal By Multi-Branch Hierarchical Segmentation

Class-Specified Segmentation with Multi-scale Superpixels

Small-scale Image Semantic Segmentation Method Based on Multi-level Superposition and Enhancement Fusion

A combined method for multi-class image semantic segmentation

Multi-Class Part Parsing with Joint Boundary-Semantic Awareness.

Semi-supervised Semantic Segmentation with Complementary Reconfirmation Mechanism.

CMF: Cascaded Multi-Model Fusion for Referring Image Segmentation

Semantic Image Segmentation Using Low-Level Features and Contextual Cues.

Research on Image Semantic Segmentation Based on Hybrid Cascade Feature Fusion and Detailed Attention Mechanism