Abstract:Video captioning aims at automatically generating descriptive sentences based on the given video, establishing an association between the visual contents and textual languages, has attracted great attention and plays a significant role in many practical applications. Previous researches focus more on the aspect of caption generation, ignoring the alignment of multimodal feature and just simply concatenating them. Besides, video feature extraction is usually done in an off-line manner, which leads to the fact that the extracted feature may not adapted to the subsequent caption generation task. To improve the applicability of extracted features for downstream caption generation and to address the issue of multimodal semantic alignment fusion, we propose an end-to-end center-enhanced video captioning model with multimodal semantic alignment, which integrates feature extraction and caption generation task into a unified framework. In order to enhance the completeness of semantic features, we design a center enhancement strategy where the visual-textual deep joint semantic feature can be captured via incremental clustering, then the cluster centers can serve as the guidance for better caption generation. Moreover, we propose to promote the visual-textual multimodal alignment fusion by learning the visual and textual representation in a shared latent semantic space, so as to alleviate the multimodal misalignment problem. Experimental results on two popular datasets MSVD and MSR-VTT demonstrate that the proposed model could outperform the state-of-the-art methods, obtaining higher-quality caption results.

Research on Feature Extraction and Multimodal Fusion of Video Caption Based on Deep Learning

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

Emotion Recognition in Videos via Fusing Multimodal Features.

Audio-Visual Speech Enhancement with Deep Multi-modality Fusion

Fusion of Multi-Modal Features to Enhance Dense Video Caption

Integrating both Visual and Audio Cues for Enhanced Video Caption

Event-centric multi-modal fusion method for dense video captioning

Scene captioning with deep fusion of images and point clouds

Attention-based Visual-Audio Fusion for Video Caption Generation.

Deep Relationship Analysis in Video with Multimodal Feature Fusion

Multimodal feature fusion based on object relation for video captioning

Center-enhanced video captioning model with multimodal semantic alignment

Multimodal Deep Representation Learning for Video Classification

Joint Learning for Relationship and Interaction Analysis in Video with Multimodal Feature Fusion

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

MFVC: Urban Traffic Scene Video Caption Based on Multimodal Fusion

Rethinking the constraints of multimodal fusion: case study in Weakly-Supervised Audio-Visual Video Parsing

Multi-modal Deep Analysis for Multimedia

A Multimodal Sentiment Analysis Approach Based on a Joint Chained Interactive Attention Mechanism

Feature Extraction Network with Attention Mechanism for Data Enhancement and Recombination Fusion for Multimodal Sentiment Analysis

Fusing Multi-Stream Deep Networks for Video Classification