Abstract:Emotional video captioning (EVC) is an emerging task to describe the factual content with the inherent emotion expressed in a video. It is crucial for the EVC task to effectively perceive subtle and ambiguous visual emotion cues in the stage of caption generation. However, existing captioning methods usually overlooked the learning of emotions in user-generated videos, thus making the generated sentence a bit boring and soulless. To address this issue, this paper proposes a new emotional captioning perspective in a human-like perception-priority manner, i.e., first perceiving the inherent emotion and then leveraging the perceived emotion cue to support caption generation. Specifically, we devise an Emotion-Prior Awareness Network (EPAN). It mainly benefits from a novel tree-structured emotion learning module involving both catalog-level psychological categories and lexical-level usual words to achieve the goal of explicit and fine-grained emotion perception. Besides, we develop a novel subordinate emotion masking mechanism between the catalog level and lexical level that facilitates coarse-to-fine emotion learning. Afterward, with the emotion prior, we can effectively decode the emotional caption by exploiting the complementation of visual, textual, and emotional semantics. In addition, we also introduce three simple yet effective optimization objectives, which can significantly boost the emotion learning from the perspectives of emotional captioning, hierarchical emotion classification, and emotional contrastive learning. Sufficient experimental results on three benchmark datasets clearly demonstrate the advantages of our proposed EPAN over existing SOTA methods in both semantic and emotional metrics. The extensive ablation study and visualization analysis further reveal the good interpretability of our emotional video captioning method. Code will be made available at https://github.com/songpipi/EPAN.

Contextual Attention Network for Emotional Video Captioning

Emotion-Prior Awareness Network for Emotional Video Captioning

Emotional Video Captioning With Vision-Based Emotion Interpretation Network

Learning Deep Contextual Attention Network for Narrative Photo Stream Captioning

MPEG: A Multi-Perspective Enhanced Graph Attention Network for Causal Emotion Entailment in Conversations

Dual-path Collaborative Generation Network for Emotional Video Captioning

Context-Aware Visual Policy Network for Fine-Grained Image Captioning

Contextual and Selective Attention Networks for Image Captioning

Context-Aware Emotion Recognition Networks

An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos

Memory-attended semantic context-aware network for video captioning

Attend to Where and When: Cascaded Attention Network for Facial Expression Recognition

Taking an Emotional Look at Video Paragraph Captioning

Context- and Sentiment-Aware Networks for Emotion Recognition in Conversation

Syntax-Guided Hierarchical Attention Network for Video Captioning

CANet: Comprehensive Attention Network for video-based action recognition

An Emotion-Embedded Visual Attention Model for Dimensional Emotion Context Learning.

Spatio-temporal Convolutional Emotional Attention Network for Spotting Macro- and Micro-Expression Intervals in Long Video Sequences

A Contextual Attention Network for Multimodal Emotion Recognition in Conversation

Image Captioning with Emotional Information Via Multiple Model

Activitynet 2019 Task 3: Exploring Contexts for Dense Captioning Events in Videos