Abstract:Recent progress in using long short-term memory (LSTM) for image captioning has motivated the exploration of their applications for video captioning. By taking a video as a sequence of features, an LSTM model is trained on video-sentence pairs and learns to associate a video to a sentence. However, most existing methods compress an entire video shot or frame into a static representation, without considering attention mechanism which allows for selecting salient features. Furthermore, existing approaches usually model the translating error, but ignore the correlations between sentence semantics and visual content. To tackle these issues, we propose a novel end-to-end framework named aLSTMs, an attention-based LSTM model with semantic consistency, to transfer videos to natural sentences. This framework integrates attention mechanism with LSTM to capture salient structures of video, and explores the correlation between multimodal representations (i.e., words and visual content) for generating sentences with rich semantic content. Specifically, we first propose an attention mechanism that uses the dynamic weighted sum of local two-dimensional convolutional neural network representations. Then, an LSTM decoder takes these visual features at time t and the word-embedding feature at time t-1 to generate important words. Finally, we use multimodal embedding to map the visual and sentence features into a joint space to guarantee the semantic consistence of the sentence description and the video visual content. Experiments on the benchmark datasets demonstrate that our method using single feature can achieve competitive or even better results than the state-of-the-art baselines for video captioning in both BLEU and METEOR.

Joint Multi-Scale Information and Long-Range Dependence for Video Captioning

Multi-scale features with temporal information guidance for video captioning

Bidirectional Long-Short Term Memory for Video Description

Richer Semantic Visual and Language Representation for Video Captioning

Rich Visual and Language Representation with Complementary Semantics for Video Captioning

Multi-guiding Long Short-Term Memory for Video Captioning

Multi-level video captioning method based on semantic space

CC-LSTM: Cross and Conditional Long-Short Time Memory for Video Captioning

Video Captioning with External Knowledge Assistance and Multi-feature Fusion

Exploiting long-term temporal dynamics for video captioning

Multirate Multimodal Video Captioning.

Learning Multimodal Attention LSTM Networks for Video Captioning.

Multi-Task Video Captioning with a Stepwise Multimodal Encoder

Multimodal Memory Modelling for Video Captioning

Enhanced Video Caption Generation Based on Multimodal Features.

Multi-Modal interpretable automatic video captioning

Exploiting the Local Temporal Information for Video Captioning.

Motion Guided Region Message Passing for Video Captioning

Video Captioning with Boundary-aware Hierarchical Language Decoding and Joint Video Prediction

Video Captioning With Attention-Based LSTM and Semantic Consistency

DCMSTRD: End-to-end Dense Captioning via Multi-Scale Transformer Decoding