Abstract:Recent progress in using long short-term memory (LSTM) for image captioning has motivated the exploration of their applications for video captioning. By taking a video as a sequence of features, an LSTM model is trained on video-sentence pairs and learns to associate a video to a sentence. However, most existing methods compress an entire video shot or frame into a static representation, without considering attention mechanism which allows for selecting salient features. Furthermore, existing approaches usually model the translating error, but ignore the correlations between sentence semantics and visual content. To tackle these issues, we propose a novel end-to-end framework named aLSTMs, an attention-based LSTM model with semantic consistency, to transfer videos to natural sentences. This framework integrates attention mechanism with LSTM to capture salient structures of video, and explores the correlation between multimodal representations (i.e., words and visual content) for generating sentences with rich semantic content. Specifically, we first propose an attention mechanism that uses the dynamic weighted sum of local two-dimensional convolutional neural network representations. Then, an LSTM decoder takes these visual features at time t and the word-embedding feature at time t-1 to generate important words. Finally, we use multimodal embedding to map the visual and sentence features into a joint space to guarantee the semantic consistence of the sentence description and the video visual content. Experiments on the benchmark datasets demonstrate that our method using single feature can achieve competitive or even better results than the state-of-the-art baselines for video captioning in both BLEU and METEOR.

CAM-RNN: Co-Attention Model Based RNN for Video Captioning.

MAM-RNN: Multi-level Attention Model Based RNN for Video Captioning.

Recurrent convolutional video captioning with global and local attention.

Memory-Attended Recurrent Network For Video Captioning

Attention based CNN-LSTM network for video caption

Dual-Stream Recurrent Neural Network for Video Captioning

Visual Attention Based on Long-Short Term Memory Model for Image Caption Generation

CC-LSTM: Cross and Conditional Long-Short Time Memory for Video Captioning

Video Captioning Via Two-stage Attention Model and Generative Adversarial Network

Visual Commonsense-Aware Representation Network for Video Captioning

Video Captioning With Attention-Based LSTM and Semantic Consistency

Non-Autoregressive Coarse-to-Fine Video Captioning

Convolutional Reconstruction-to-Sequence for Video Captioning

Recurrent Image Captioner: Describing Images with Spatial-Invariant Transformation and Attention Filtering

Learning Multimodal Attention LSTM Networks for Video Captioning.

From Deterministic to Generative: Multimodal Stochastic RNNs for Video Captioning

Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Bidirectional Multimodal Recurrent Neural Networks with Refined Visual Features for Image Captioning.

Boundary Detector Encoder And Decoder With Soft Attention For Video Captioning

Multi-guiding Long Short-Term Memory for Video Captioning

From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning