Abstract:Video captioning aims to automatically describe a video clip with informative sentences. At present, deep learning-based models have become the mainstream for this task and achieved competitive results on public datasets. Usually, these methods leverage different types of features to generate sentences, e.g., semantic information, 2D or 3D features. However, some methods only treat semantic information as a complement of visual representations and cannot fully exploit it; some of them ignore the relationship between different types of features. In addition, most of them select multiple frames of a video with an equally spaced sampling scheme, resulting in much redundant information. To address these issues, we present a novel video-captioning framework, Semantic Enhanced video captioning with Multi-feature Fusion, SEMF for short. It optimizes the use of different types of features from three aspects. First, a semantic encoder is designed to enhance meaningful semantic features through a semantic dictionary to boost performance. Second, a discrete selection module pays attention to important features and obtains different contexts at different steps to reduce feature redundancy. Finally, a multi-feature fusionmodule uses a novel relation-aware attentionmechanism to separate the common and complementary components of different features to provide more effective visual features for the next step. Moreover, the entire framework can be trained in an end-to-endmanner. Extensive experiments are conducted on Microsoft Research Video Description Corpus (MSVD) and MSR-Video to Text (MSR-VTT) datasets. The results demonstrate that SEMF is able to achieve state-of-the-art results.

Hierarchical Attention-Based Multimodal Fusion for Video Captioning

Multimodal-enhanced hierarchical attention network for video captioning

Attention-based Visual-Audio Fusion for Video Caption Generation.

Attention-Based Multimodal Fusion for Video Description

Multimodal Semantic Attention Network for Video Captioning

Fusion of Multi-Modal Features to Enhance Dense Video Caption

Research on Video Captioning Based on Multifeature Fusion.

Semantic Enhanced Video Captioning with Multi-feature Fusion

Hierarchical & Multimodal Video Captioning: Discovering and Transferring Multimodal Knowledge for Vision to Language

Learning Multimodal Attention LSTM Networks for Video Captioning.

Event-centric multi-modal fusion method for dense video captioning

Integrating both Visual and Audio Cues for Enhanced Video Caption

Research on Feature Extraction and Multimodal Fusion of Video Caption Based on Deep Learning

Semantic Embedding Guided Attention with Explicit Visual Feature Fusion for Video Captioning

A Multimodal Sentiment Analysis Approach Based on a Joint Chained Interactive Attention Mechanism

Enhanced Video Caption Generation Based on Multimodal Features.

Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning

Center-enhanced video captioning model with multimodal semantic alignment

Multimodal Sentiment Analysis Based on Composite Hierarchical Fusion

Bi-attention Modal Separation Network for Multimodal Video Fusion

Describing Videos Using Multi-modal Fusion.