Abstract:Image Captioning, which automatically describes an image with natural language, is regarded as a fundamental challenge in computer vision. In recent years, significant advance has been made in image captioning through improving attention mechanism. However, most existing methods construct attention mechanisms based on singular visual features, such as patch features or object features, which limits the accuracy of generated captions. In this article, we propose a Bidirectional Co-Attention Network (BCAN) that combines multiple visual features to provide information from different aspects. Different features are associated with predicting different words, and there are a priori relations between these multiple visual features. Based on this, we further propose a bottom-up and top-down bi-directional co-attention mechanism to extract discriminative attention information. Furthermore, most existing methods do not exploit an effective multimodal integration strategy, generally using addition or concatenation to combine features. To solve this problem, we adopt the Multivariate Residual Module (MRM) to integrate multimodal attention features. Meanwhile, we further propose a Vertical MRM to integrate features of the same category, and a Horizontal MRM to combine features of the different categories, which can balance the contribution of the bottom-up co-attention and the top-down co-attention. In contrast to the existing methods, the BCAN is able to obtain complementary information from multiple visual features via the bi-directional co-attention strategy, and integrate multimodal information via the improved multivariate residual strategy. We conduct a series of experiments on two benchmark datasets (MSCOCO and Flickr30k), and the results indicate that the proposed BCAN achieves the superior performance.

Hierarchical Deep Neural Network for Image Captioning

A Hierarchical Multimodal Attention-based Neural Network for Image Captioning

Better Understanding Hierarchical Visual Relationship for Image Caption

Positional Self-attention Based Hierarchical Image Captioning.

Deep Hierarchical Attention Network for Video Description

Semantic Guided Level-Category Hybrid Prediction Network for Hierarchical Image Classification.

Boosting convolutional image captioning with semantic content and visual relationship

Phrase-based Image Captioning with Hierarchical LSTM Model

Multi-modal reward for visual relationships-based image captioning

Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks

Explainable Image Captioning using CNN- CNN architecture and Hierarchical Attention

Neural Symbolic Representation Learning for Image Captioning

Tell and Guess: Cooperative Learning for Natural Image Caption Generation with Hierarchical Refined Attention

Recurrent Image Captioner: Describing Images with Spatial-Invariant Transformation and Attention Filtering

Deep Semantic Understanding of High Resolution Remote Sensing Image

Image Captioning with Deep Bidirectional LSTMs

Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning

IMAGE CAPTION BASED ON IMAGE SEMANTIC SIMILARITY NETWORK

Enhanced Image Captioning with Color Recognition Using Deep Learning Methods

Hierarchical LSTMs with Adaptive Attention for Visual Captioning

Bi-Directional Co-Attention Network for Image Captioning