Abstract:Recent progress in using long short-term memory (LSTM) for image captioning has motivated the exploration of their applications for video captioning. By taking a video as a sequence of features, an LSTM model is trained on video-sentence pairs and learns to associate a video to a sentence. However, most existing methods compress an entire video shot or frame into a static representation, without considering attention mechanism which allows for selecting salient features. Furthermore, existing approaches usually model the translating error, but ignore the correlations between sentence semantics and visual content. To tackle these issues, we propose a novel end-to-end framework named aLSTMs, an attention-based LSTM model with semantic consistency, to transfer videos to natural sentences. This framework integrates attention mechanism with LSTM to capture salient structures of video, and explores the correlation between multimodal representations (i.e., words and visual content) for generating sentences with rich semantic content. Specifically, we first propose an attention mechanism that uses the dynamic weighted sum of local two-dimensional convolutional neural network representations. Then, an LSTM decoder takes these visual features at time t and the word-embedding feature at time t-1 to generate important words. Finally, we use multimodal embedding to map the visual and sentence features into a joint space to guarantee the semantic consistence of the sentence description and the video visual content. Experiments on the benchmark datasets demonstrate that our method using single feature can achieve competitive or even better results than the state-of-the-art baselines for video captioning in both BLEU and METEOR.

Enhancing the Alignment Between Target Words and Corresponding Frames for Video Captioning.

SBAT: Video Captioning with Sparse Boundary-Aware Transformer

Learning Video-Text Aligned Representations for Video Captioning

Align and Tell: Boosting Text-Video Retrieval With Local Alignment and Fine-Grained Supervision

Measuring apoptosis in neural stem cells.

TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification

Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning

STAT: Spatial-Temporal Attention Mechanism for Video Captioning

MTA: Multimodal Task Alignment for BEV Perception and Captioning

Self Attention Re-encoding and Linguistic Ability Preserving for Context-Aware Video Captioning

Progress-Aware Video Frame Captioning

FTAN: Exploring Frame-Text Attention for Lightweight Video Captioning.

Video Captioning With Attention-Based LSTM and Semantic Consistency

Motion Guided Region Message Passing for Video Captioning

ALCAP: Alignment-Augmented Music Captioner

Delving Deeper into the Decoder for Video Captioning

Fused GRU with Semantic-Temporal Attention for Video Captioning.

Integrating both Visual and Audio Cues for Enhanced Video Caption

Non-Autoregressive Coarse-to-Fine Video Captioning

Object Relational Graph with Teacher-Recommended Learning for Video Captioning