Abstract:Image captioning has become a hot topic in artificial intelligence research and sits at the intersection of computer vision and natural language processing. Most recent imaging captioning models have adopted an “encoder + decoder” architecture, in which the encoder is employed generally to extract the visual feature, while the decoder generates the descriptive sentence word by word. However, the visual features need to be flattened into sequence form before being forwarded to the decoder, and this results in the loss of the 2D spatial position information of the image. This limitation is particularly pronounced in the Transformer architecture since it is inherently not position-aware. Therefore, in this paper, we propose a simple coordinate-based spatial position encoding method (CSPE) to remedy this deficiency. CSPE firstly creates the 2D position coordinates for each feature pixel, and then encodes them by row and by column separately via trainable or hard encoding, effectively strengthening the position representation of visual features and enriching the generated description sentences. In addition, in order to reduce the time cost, we also explore a diagonal-based spatial position encoding (DSPE) approach. Compared with CSPE, DSPE is slightly inferior in performance but has a faster calculation speed. Extensive experiments on the MS COCO 2014 dataset demonstrate that CSPE and DSPE can significantly enhance the spatial position representation of visual features. CSPE, in particular, demonstrates BLEU-4 and CIDEr metrics improved by 1.6% and 5.7%, respectively, compared with a baseline model without sequence-based position encoding, and also outperforms current sequence-based position encoding approaches by a significant margin. In addition, the robustness and plug-and-play ability of the proposed method are validated based on a medical captioning generation model.

GridPE: Unifying Positional Encoding in Transformers with a Grid Cell-Inspired Framework

Grid-augumented vision: A simple yet effective approach for enhanced spatial understanding in multi-modal agents

Why Grid Cells Function As a Metric for Space.

Exploring Spatial-Based Position Encoding for Image Captioning

Rethinking and Improving Relative Position Encoding for Vision Transformer

Learning positional encodings in transformers depends on initialization

Explore Better Relative Position Embeddings from Encoding Perspective for Transformer Models.

Conditional Positional Encodings for Vision Transformers

Grid Cell Modeling with Mapping Representation of Self-Motion for Path Integration

Comparing Graph Transformers via Positional Encodings

Complex-Valued Relative Positional Encodings for Transformer

Algebraic Positional Encodings

A bio-inspired positional embedding network for transformer-based models

A Simple and Effective Positional Encoding for Transformers

The Impact of Positional Encoding on Length Generalization in Transformers

HyPE-GT: where Graph Transformers meet Hyperbolic Positional Encodings

Co-ordinate-based positional embedding that captures resolution to enhance transformer's performance in medical image analysis

Bridging Graph Position Encodings for Transformers with Weighted Graph-Walking Automata

Your Transformer May Not be as Powerful as You Expect

Memory Positional Encoding for Image Captioning

Relative Positional Encoding Family via Unitary Transformation