Abstract:Image captioning has become a hot topic in artificial intelligence research and sits at the intersection of computer vision and natural language processing. Most recent imaging captioning models have adopted an “encoder + decoder” architecture, in which the encoder is employed generally to extract the visual feature, while the decoder generates the descriptive sentence word by word. However, the visual features need to be flattened into sequence form before being forwarded to the decoder, and this results in the loss of the 2D spatial position information of the image. This limitation is particularly pronounced in the Transformer architecture since it is inherently not position-aware. Therefore, in this paper, we propose a simple coordinate-based spatial position encoding method (CSPE) to remedy this deficiency. CSPE firstly creates the 2D position coordinates for each feature pixel, and then encodes them by row and by column separately via trainable or hard encoding, effectively strengthening the position representation of visual features and enriching the generated description sentences. In addition, in order to reduce the time cost, we also explore a diagonal-based spatial position encoding (DSPE) approach. Compared with CSPE, DSPE is slightly inferior in performance but has a faster calculation speed. Extensive experiments on the MS COCO 2014 dataset demonstrate that CSPE and DSPE can significantly enhance the spatial position representation of visual features. CSPE, in particular, demonstrates BLEU-4 and CIDEr metrics improved by 1.6% and 5.7%, respectively, compared with a baseline model without sequence-based position encoding, and also outperforms current sequence-based position encoding approaches by a significant margin. In addition, the robustness and plug-and-play ability of the proposed method are validated based on a medical captioning generation model.

Complex-Valued Relative Positional Encodings for Transformer

Relative Positional Encoding Family via Unitary Transformation

On Scalar Embedding of Relative Positions in Attention Models

Rethinking and Improving Relative Position Encoding for Vision Transformer

Explore Better Relative Position Embeddings from Encoding Perspective for Transformer Models.

RoFormer: Enhanced Transformer with Rotary Position Embedding

Improve Transformer Models with Better Relative Position Embeddings

Linearized Relative Positional Encoding

Bidirectional Transformer with Absolute-Position Aware Relative Position Encoding for Encoding Sentences

Conditional Positional Encodings for Vision Transformers

A Simple and Effective Positional Encoding for Transformers

Your Transformer May Not be as Powerful as You Expect

EulerFormer: Sequential User Behavior Modeling with Complex Vector Attention

Learning positional encodings in transformers depends on initialization

An Empirical Study on the Impact of Positional Encoding in Transformer-based Monaural Speech Enhancement

Design of a Modified Transformer Architecture Based on Relative Position Coding

Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings

Exploring Spatial-Based Position Encoding for Image Captioning

Rethinking Position Embedding Methods in the Transformer Architecture

GridPE: Unifying Positional Encoding in Transformers with a Grid Cell-Inspired Framework

GRPE: Relative Positional Encoding for Graph Transformer