Abstract:Effectively summarizing and re-expressing video content by natural languages in a more human-like fashion is one of the key topics in the field of multimedia content understanding. Despite good progress made in recent years, existing efforts usually overlooked the emotions in user-generated videos, thus making the generated sentence a bit boring and soulless. To fill the research gap, this paper presents a novel emotional video captioning framework in which we design a Vision-based Emotion Interpretation Network to effectively capture the emotions conveyed in videos and describe the visual content in both factual and emotional languages. Specifically, we first model the emotion distribution over an open psychological vocabulary to predict the emotional state of videos. Then, guided by the discovered emotional state, we incorporate visual context, textual context, and visual-textual relevance into an aggregated multimodal contextual vector to enhance video captioning. Furthermore, we optimize the network in a new emotion-fact coordinated way that involves two losses— Emotional Indication Loss and Factual Contrastive Loss, which penalize the error of emotion prediction and visual-textual factual relevance, respectively. In other words, we innovatively introduce emotional representation learning into an end-to-end video captioning network. Extensive experiments on public benchmark datasets, EmVidCap and EmVidCap-S, demonstrate that our method can significantly outperform the state-of-the-art methods by a large margin. Quantitative ablation studies and qualitative analyses clearly show that our method is able to effectively capture the emotions in videos and thus generate emotional language sentences to interpret the video content.

Personalized Video Emotion Tagging Through a Topic Model

Content-Based Video Emotion Tagging Augmented by Users’ Multiple Physiological Responses

Implicit Hybrid Video Emotion Tagging By Integrating Video Content And Users' Multiple Physiological Responses

Emotional Tagging Of Videos By Exploring Multiple Emotions' Coexistence

Tracking the evolution of social emotions with topic models

Implicit Video Emotion Tagging from Audiences’ Facial Expression

Capturing Dependencies among Labels and Features for Multiple Emotion Tagging of Multimedia Data.

Video emotional description with fact reinforcement and emotion awaking

Multiple Emotional Tagging of Multimedia Data by Exploiting Dependencies among Emotions.

Exploiting Multi-Expression Dependences for Implicit Multi-Emotion Video Tagging.

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

Emotional Video Captioning With Vision-Based Emotion Interpretation Network

A Blast From the Past: Personalizing Predictions of Video-Induced Emotions using Personal Memories as Context

Video emotion analysis enhanced by recognizing emotion in video comments

Implicit Video Multi-Emotion Tagging by Exploiting Multi-Expression Relations.

Hybrid video emotional tagging using users’ EEG and video content

Implicit Media Tagging and Affect Prediction from video of spontaneous facial expressions, recorded with depth camera

Multimodal Topic Learning for Video Recommendation

Predicting Emotions in User-Generated Videos.

How Would The Viewer Feel? Estimating Wellbeing From Video Scenarios

A probabilistic graphical model for topic and preference discovery on social media