Abstract:Image Captioning, which automatically describes an image with natural language, is regarded as a fundamental challenge in computer vision. In recent years, significant advance has been made in image captioning through improving attention mechanism. However, most existing methods construct attention mechanisms based on singular visual features, such as patch features or object features, which limits the accuracy of generated captions. In this article, we propose a Bidirectional Co-Attention Network (BCAN) that combines multiple visual features to provide information from different aspects. Different features are associated with predicting different words, and there are a priori relations between these multiple visual features. Based on this, we further propose a bottom-up and top-down bi-directional co-attention mechanism to extract discriminative attention information. Furthermore, most existing methods do not exploit an effective multimodal integration strategy, generally using addition or concatenation to combine features. To solve this problem, we adopt the Multivariate Residual Module (MRM) to integrate multimodal attention features. Meanwhile, we further propose a Vertical MRM to integrate features of the same category, and a Horizontal MRM to combine features of the different categories, which can balance the contribution of the bottom-up co-attention and the top-down co-attention. In contrast to the existing methods, the BCAN is able to obtain complementary information from multiple visual features via the bi-directional co-attention strategy, and integrate multimodal information via the improved multivariate residual strategy. We conduct a series of experiments on two benchmark datasets (MSCOCO and Flickr30k), and the results indicate that the proposed BCAN achieves the superior performance.

Learning Deep Contextual Attention Network for Narrative Photo Stream Captioning

Contextual and Selective Attention Networks for Image Captioning

Contextual Attention Network for Emotional Video Captioning

Let Your Photos Talk: Generating Narrative Paragraph For Photo Stream Via Bidirectional Attention Recurrent Neural Networks

Context-Aware Visual Policy Network for Fine-Grained Image Captioning

Triple-stream Commonsense Circulation Transformer Network for Image Captioning

Looking Deeper and Transferring Attention for Image Captioning.

A Dual Self-Attention based Network for Image Captioning

CA-Captioner: A Novel Concentrated Attention for Image Captioning

Bi-Directional Co-Attention Network for Image Captioning

Learning visual relationship and context-aware attention for image captioning

Show, Reward and Tell: Automatic Generation of Narrative Paragraph from Photo Stream by Adversarial Training

Think As People: Context-Driven Multi-Image News Captioning with Adaptive Dual Attention

Context-Aware Visual Policy Network for Sequence-Level Image Captioning

A Sequential Guiding Network with Attention for Image Captioning

Deliberate Attention Networks for Image Captioning

Exploring Overall Contextual Information for Image Captioning in Human-Like Cognitive Style

Hybrid Attention Network for Image Captioning

Hierarchical Attention Network for Image Captioning.

Spatio-Temporal Graph-based Semantic Compositional Network for Video Captioning

CaptionNet: A Tailor-made Recurrent Neural Network for Generating Image Descriptions