Abstract:For remote sensing image (RSI) captioning tasks, two-stage RSI captioning methods have achieved high performance because they introduce the results of other RSI tasks, such as image classification as prior information. However, most previous works treat image classification in the two-stage RSI captioning method as a single-label classification task, which cannot adequately describe the entire contents of complex RSIs and may cause semantic ambiguity. To settle this problem and further refine image feature representation, we introduce multi-label classification into two-stage RSI captioning to provide sufficient and accurate prior semantic information and propose a multi-label semantic feature fusion (MLSFF) framework. Specifically, we design a robust multi-label semantic attribute extractor to extract multi-label semantic attributes of RSIs. To obtain discriminative feature representations for RSI captioning, we propose two cross-modal semantic feature fusion operators that fuse the extracted semantic attributes and the image feature extracted by the convolutional neural network. The results of extensive numerical experiments show that the proposed method can achieve state-of-the-art performance on the UCM-Captions, Sydney-Captions, and RSICD datasets. Specifically, on the UCM-Captions dataset, our method achieves a gain of 8.2% in Sm score over the SAT (LAM) method (Zhang et al., 2019c). On the Sydney-Captions dataset, our method improves the Sm score by 17.4% compared with the TCE loss-based method (Li et al., 2020a). On the RSICD, our method outperforms the multi-level attention method (Li et al., 2020b) by 3.2% in terms of the S-m score. Code is available at https://github.com/xtye5025/MLSFF.

Research on Video Captioning Based on Multifeature Fusion.

Semantic Enhanced Video Captioning with Multi-feature Fusion

Integrating both Visual and Audio Cues for Enhanced Video Caption

Fusion of Multi-Modal Features to Enhance Dense Video Caption

Research on Feature Extraction and Multimodal Fusion of Video Caption Based on Deep Learning

Center-enhanced video captioning model with multimodal semantic alignment

Video Captioning with External Knowledge Assistance and Multi-feature Fusion

Enhanced Video Caption Generation Based on Multimodal Features.

Hierarchical Attention-Based Multimodal Fusion for Video Captioning

Semantic Embedding Guided Attention with Explicit Visual Feature Fusion for Video Captioning

Multi-scale features with temporal information guidance for video captioning

MFVC: Urban Traffic Scene Video Caption Based on Multimodal Fusion

Attention-based Visual-Audio Fusion for Video Caption Generation.

Multi-Task Video Captioning with a Stepwise Multimodal Encoder

ACF-net: Appearance-Guided Content Filter Network for Video Captioning

Multi-label Semantic Feature Fusion for Remote Sensing Image Captioning

Multimodal feature fusion based on object relation for video captioning

Multimodality-guided Visual-Caption Semantic Enhancement

Multi-View Feature Fusion and Visual Prompt for Remote Sensing Image Captioning

Global-Shared Text Representation Based Multi-Stage Fusion Transformer Network for Multi-Modal Dense Video Captioning

Event-centric multi-modal fusion method for dense video captioning