Abstract:Image captioning has become a hot topic in artificial intelligence research and sits at the intersection of computer vision and natural language processing. Most recent imaging captioning models have adopted an “encoder + decoder” architecture, in which the encoder is employed generally to extract the visual feature, while the decoder generates the descriptive sentence word by word. However, the visual features need to be flattened into sequence form before being forwarded to the decoder, and this results in the loss of the 2D spatial position information of the image. This limitation is particularly pronounced in the Transformer architecture since it is inherently not position-aware. Therefore, in this paper, we propose a simple coordinate-based spatial position encoding method (CSPE) to remedy this deficiency. CSPE firstly creates the 2D position coordinates for each feature pixel, and then encodes them by row and by column separately via trainable or hard encoding, effectively strengthening the position representation of visual features and enriching the generated description sentences. In addition, in order to reduce the time cost, we also explore a diagonal-based spatial position encoding (DSPE) approach. Compared with CSPE, DSPE is slightly inferior in performance but has a faster calculation speed. Extensive experiments on the MS COCO 2014 dataset demonstrate that CSPE and DSPE can significantly enhance the spatial position representation of visual features. CSPE, in particular, demonstrates BLEU-4 and CIDEr metrics improved by 1.6% and 5.7%, respectively, compared with a baseline model without sequence-based position encoding, and also outperforms current sequence-based position encoding approaches by a significant margin. In addition, the robustness and plug-and-play ability of the proposed method are validated based on a medical captioning generation model.

Multi-Scale Cropping Mechanism for Remote Sensing Image Captioning

Exploring Multi-Level Attention and Semantic Relationship for Remote Sensing Image Captioning

Multiscale Methods for Optical Remote-Sensing Image Captioning

Remote Sensing Image Captioning Based on Multi-Level Feature Extraction and Adaptive Attention

Remote Sensing Image Captioning with Multi-Scale Feature and Small Target Attention

Denoising-Based Multiscale Feature Fusion for Remote Sensing Image Captioning.

High-Resolution Remote Sensing Image Captioning Based on Structured Attention

Multi-Scale Remote Sensing Semantic Analysis Based On A Global Perspective

INSTANCE-AWARE REMOTE SENSING IMAGE CAPTIONING WITH CROSS-HIERARCHY ATTENTION

A Patch-Level Region-Aware Module with a Multi-Label Framework for Remote Sensing Image Captioning

Remote-Sensing Image Captioning Based on Multilayer Aggregated Transformer

Remote Sensing Image Captioning with Sequential Attention and Flexible Word Correlation

Exploring Spatial-Based Position Encoding for Image Captioning

A Multi-Level Attention Model For Remote Sensing Image Captions

TSFE: Two-Stage Feature Enhancement for Remote Sensing Image Captioning

Cross-Modal Retrieval and Semantic Refinement for Remote Sensing Image Captioning

Exploring Models and Data for Remote Sensing Image Caption Generation

Meta Captioning: A Meta Learning Based Remote Sensing Image Captioning Framework

A Joint-Training Two-Stage Method For Remote Sensing Image Captioning.

Semantic Descriptions of High-Resolution Remote Sensing Images

Remote Sensing Image Caption Generation Via Transformer and Reinforcement Learning.