Abstract:Self-attention mechanism, which has been successfully applied to current encoder-decoder framework of image captioning, is used to enhance the feature representation in the image encoder and capture the most relevant information for the language decoder. However, most existing methods will assign attention weights to all candidate vectors, which implicitly hypothesizes that all vectors are relevant. Moreover, current self-attention mechanisms ignore the intra-object attention distribution, and only consider the inter-object relationships. In this paper, we propose a Multi-Gate Attention (MGA) block, which expands the traditional self-attention by equipping with additional Attention Weight Gate (AWG) module and Self-Gated (SG) module. The former constrains the attention weights to be assigned to the most contributive objects. The latter is adopted to consider the intra-object attention distribution and eliminate the irrelevant information in object feature vector. Furthermore, most current image captioning methods apply the original transformer designed for natural language processing task, to refine image features directly. Therefore, we propose a pre-layernorm transformer to simplify the transformer architecture and make it more efficient for image feature enhancement. By integrating MGA block with pre-layernorm transformer architecture into the image encoder and AWG module into the language decoder, we present a novel Multi-Gate Attention Network (MGAN). The experiments on MS COCO dataset indicate that the MGAN outperforms most of the state-of-the-art, and further experiments on other methods combined with MGA blocks demonstrate the generalizability of our proposal.

Graph Structural Attention and Increased Global Attention for Image Captioning

Learning visual relationship and context-aware attention for image captioning

Scene Graph Captioner: Image Captioning Based on Structural Visual Representation

A Multi-stage Feature Enhancement Network with Second-Attention for Image Captioning

Image Captioning Using DenseNet Network and Adaptive Attention.

Scene-based Factored Attention for Image Captioning

An Image Captioning Algorithm Based on Combination Attention Mechanism

Visual Attention Based on Long-Short Term Memory Model for Image Caption Generation

INSTANCE-AWARE REMOTE SENSING IMAGE CAPTIONING WITH CROSS-HIERARCHY ATTENTION

Local-global Visual Interaction Attention for Image Captioning

Multi-Gate Attention Network for Image Captioning

Exploring Visual Relationship for Image Captioning

Image Caption with Global-Local Attention

Spatial Relational Attention Using Fully Convolutional Networks for Image Caption Generation

GateCap: Gated Spatial and Semantic Attention Model for Image Captioning

Cascade Attention: Multiple Feature Based Learning for Image Captioning

Topic Scene Graph Generation by Attention Distillation from Caption

Improve Image Captioning By Self-Attention

Scene Attention Mechanism For Remote Sensing Image Caption Generation

Looking Deeper and Transferring Attention for Image Captioning.

Image Caption Based on Visual Attention Mechanism