Abstract:The task of multi-label image classification involves recognizing multiple objects within a single image. Considering both valuable semantic information contained in the labels and essential visual features presented in the image, tight visual-linguistic interactions play a vital role in improving classification performance. Moreover, given the potential variance in object size and appearance within a single image, attention to features of different scales can help to discover possible objects in the image. Recently, Transformer-based methods have achieved great success in multi-label image classification by leveraging the advantage of modeling long-range dependencies, but they have several limitations. Firstly, existing methods treat visual feature extraction and cross-modal fusion as separate steps, resulting in insufficient visual-linguistic alignment in the joint semantic space. Additionally, they only extract visual features and perform cross-modal fusion at a single scale, neglecting objects with different characteristics. To address these issues, we propose a Hierarchical Scale-Aware Vision-Language Transformer (HSVLT) with two appealing designs: (1)~A hierarchical multi-scale architecture that involves a Cross-Scale Aggregation module, which leverages joint multi-modal features extracted from multiple scales to recognize objects of varying sizes and appearances in images. (2)~Interactive Visual-Linguistic Attention, a novel attention mechanism module that tightly integrates cross-modal interaction, enabling the joint updating of visual, linguistic and multi-modal features. We have evaluated our method on three benchmark datasets. The experimental results demonstrate that HSVLT surpasses state-of-the-art methods with lower computational cost.

STMG: Swin transformer for multi-label image recognition with graph convolution network

SLViT: Scale-Wise Language-Guided Vision Transformer for Referring Image Segmentation.

Asymmetric Vision Transformers for Multi-Label Classification

Graph Attention Transformer Network for Multi-Label Image Classification

NGST-Net: A N-Gram based Swin Transformer Network for improving multispectral and hyperspectral image fusion

Class-Guided Swin Transformer for Semantic Segmentation of Remote Sensing Imagery

A multimodal hyper-fusion transformer for remote sensing image classification

GA-Based Weighted Ensemble Learning for Multi-Label Aerial Image Classification Using Convolutional Neural Networks and Vision Transformers

A Spatial–Spectral Transformer for Hyperspectral Image Classification Based on Global Dependencies of Multi-Scale Features

Multi-Label Image Recognition With Graph Convolutional Networks

SST: Spatial and Semantic Transformers for Multi-Label Image Recognition

Query2Label: A Simple Transformer Way to Multi-Label Classification

Diverse Instance Discovery: Vision-Transformer for Instance-Aware Multi-Label Image Recognition.

Multi-input trademark element recognition with transformer

HSVLT: Hierarchical Scale-Aware Vision-Language Transformer for Multi-Label Image Classification

Diverse Instance Discovery: Vision-Transformer for Instance-Aware Multi-Label Image Recognition

Learning Graph Convolutional Networks for Multi-Label Recognition and Applications

TransFG: A Transformer Architecture for Fine-Grained Recognition

GKGNet: Group K-Nearest Neighbor based Graph Convolutional Network for Multi-Label Image Recognition