Abstract:In the field of computer vision, it is a challenging task to generate natural language captions from videos as input. To deal with this task, videos are usually regarded as feature sequences and input into Long-Short Term Memory (LSTM) to generate natural language. To get richer and more detailed video content representation, a Multimodal Interaction Video Captioning Network based on Semantic Association Graph (MIVCN) is developed towards this task. This network consists of two modules: Semantic association Graph Module (SAGM) and Multimodal Attention Constraint Module (MACM). Firstly, owing to lack of the semantic interdependence, existing methods often produce illogical sentence structures. Therefore, we propose a SAGM based on information association, which enables network to strengthen the connection between logically related languages and alienate the relations between logically unrelated languages. Secondly, features of each modality need to pay attention to different information among them, and the captured multimodal features are great informative and redundant. Based on the discovery, we propose a MACM based on LSTM, which can capture complementary visual features and filter redundant visual features. The MACM is applied to integrate multimodal features into LSTM, and make network to screen and focus on informative features. Through the association of semantic attributes and the interaction of multimodal features, the semantically contextual interdependent and visually complementary information can be captured by this network, and the informative representation in videos also can be better used for generating captioning. The proposed MIVCN realizes the best caption generation performance on MSVD: 56.8%, 36.4%, and 79.1% on BLEU@4, METEOR, and ROUGE-L evaluation metrics, respectively. Superior results are also reported on MSR-VTT about BLEU@4, METEOR, and ROUGE-L compared to state-of-the-art methods.

MAM-RNN: Multi-level Attention Model Based RNN for Video Captioning.

CAM-RNN: Co-Attention Model Based RNN for Video Captioning.

Recurrent convolutional video captioning with global and local attention.

Memory-Attended Recurrent Network For Video Captioning

Visual Attention Based on Long-Short Term Memory Model for Image Caption Generation

Attention based CNN-LSTM network for video caption

Multi-guiding Long Short-Term Memory for Video Captioning

Learning Multimodal Attention LSTM Networks for Video Captioning.

Dual-Stream Recurrent Neural Network for Video Captioning

Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Video Captioning With Attention-Based LSTM and Semantic Consistency

Multimodal Memory Modelling for Video Captioning

MIVCN: Multimodal interaction video captioning network based on semantic association graph

Multimodal Semantic Attention Network for Video Captioning

Video Captioning Via Two-stage Attention Model and Generative Adversarial Network

Bidirectional Multimodal Recurrent Neural Networks with Refined Visual Features for Image Captioning.

Attention-based LSTM with Semantic Consistency for Videos Captioning

Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks

From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning

CC-LSTM: Cross and Conditional Long-Short Time Memory for Video Captioning

Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning