Abstract:Image Captioning, which automatically describes an image with natural language, is regarded as a fundamental challenge in computer vision. In recent years, significant advance has been made in image captioning through improving attention mechanism. However, most existing methods construct attention mechanisms based on singular visual features, such as patch features or object features, which limits the accuracy of generated captions. In this article, we propose a Bidirectional Co-Attention Network (BCAN) that combines multiple visual features to provide information from different aspects. Different features are associated with predicting different words, and there are a priori relations between these multiple visual features. Based on this, we further propose a bottom-up and top-down bi-directional co-attention mechanism to extract discriminative attention information. Furthermore, most existing methods do not exploit an effective multimodal integration strategy, generally using addition or concatenation to combine features. To solve this problem, we adopt the Multivariate Residual Module (MRM) to integrate multimodal attention features. Meanwhile, we further propose a Vertical MRM to integrate features of the same category, and a Horizontal MRM to combine features of the different categories, which can balance the contribution of the bottom-up co-attention and the top-down co-attention. In contrast to the existing methods, the BCAN is able to obtain complementary information from multiple visual features via the bi-directional co-attention strategy, and integrate multimodal information via the improved multivariate residual strategy. We conduct a series of experiments on two benchmark datasets (MSCOCO and Flickr30k), and the results indicate that the proposed BCAN achieves the superior performance.

Multivariate Attention Network For Image Captioning

Multi-View Attention Network for Remote Sensing Image Captioning

Bi-Directional Co-Attention Network for Image Captioning

Visual Attention Based on Long-Short Term Memory Model for Image Caption Generation

Hybrid Attention Network for Image Captioning

Hierarchical Attention Network for Image Captioning.

MAENet: A Novel Multi-Head Association Attention Enhancement Network for Completing Intra-Modal Interaction in Image Captioning

Multi-view Attention with Memory Assistant for Image Captioning

A Hierarchical Multimodal Attention-based Neural Network for Image Captioning

Multimodal Semantic Attention Network for Video Captioning

Multi-Gate Attention Network for Image Captioning

Multi-Keys Attention Network for Image Captioning

Attend to Knowledge: Memory-Enhanced Attention Network for Image Captioning.

A Multi-stage Feature Enhancement Network with Second-Attention for Image Captioning

Multi-Attention Generative Adversarial Network for Image Captioning.

Multilevel Attention Networks and Policy Reinforcement Learning for Image Caption Generation

Combining Object-Based Attention And Attributes For Image Captioning

Learning joint relationship attention network for image captioning

Improve Image Captioning By Self-Attention

Avtmnet: Adaptive Visual-Text Merging Network for Image Captioning

Multimodal-enhanced hierarchical attention network for video captioning