Abstract:Effectively summarizing and re-expressing video content by natural languages in a more human-like fashion is one of the key topics in the field of multimedia content understanding. Despite good progress made in recent years, existing efforts usually overlooked the emotions in user-generated videos, thus making the generated sentence a bit boring and soulless. To fill the research gap, this paper presents a novel emotional video captioning framework in which we design a Vision-based Emotion Interpretation Network to effectively capture the emotions conveyed in videos and describe the visual content in both factual and emotional languages. Specifically, we first model the emotion distribution over an open psychological vocabulary to predict the emotional state of videos. Then, guided by the discovered emotional state, we incorporate visual context, textual context, and visual-textual relevance into an aggregated multimodal contextual vector to enhance video captioning. Furthermore, we optimize the network in a new emotion-fact coordinated way that involves two losses— Emotional Indication Loss and Factual Contrastive Loss, which penalize the error of emotion prediction and visual-textual factual relevance, respectively. In other words, we innovatively introduce emotional representation learning into an end-to-end video captioning network. Extensive experiments on public benchmark datasets, EmVidCap and EmVidCap-S, demonstrate that our method can significantly outperform the state-of-the-art methods by a large margin. Quantitative ablation studies and qualitative analyses clearly show that our method is able to effectively capture the emotions in videos and thus generate emotional language sentences to interpret the video content.

Encoding Syntactic Dependency and Topical Information for Social Emotion Classification

A Hybrid Semantic-Topic Co-encoding Network for Social Emotion Classification

Tree-Structured Neural Networks with Topic Attention for Social Emotion Classification

An Interpretable Neural Network with Topical Information for Relevant Emotion Ranking.

An End-to-end Topic-Enhanced Self-Attention Network for Social Emotion Classification.

Topic-Enhanced Capsule Network for Multi-Label Emotion Classification.

Emotion Classification Based on Structured Information

Hierarchical Semantic Representations of Online News Comments for Emotion Tagging Using Multiple Information Sources

Hierarchical state recurrent neural network for social emotion ranking

FSS-GCN: A graph convolutional networks with fusion of semantic and structure for emotion cause analysis

Multi-document Influence on Readers: Augmenting Social Emotion Prediction by Learning Document Interactions

Lexicon-based Conversion Strategy Words Sentence Single Label Joy : 0 Fear : 0 Anger : 0 Surprise : 0 Sadness : 1

Encoding Syntactic Knowledge in Neural Networks for Sentiment Classification.

Hidden Topic-Emotion Transition Model for Multi-Level Social Emotion Detection

Detect the Emotions of the Public Based on Cascade Neural Network Model

Predicting Social Emotions from Readers’ Perspective

Multi-Source Emotion Tagging for Online News Comments Using Bi-Directional Hierarchical Semantic Representation Model

Senti ment Key Sentence Identification Based on LexicaI Semantics and Syntactic Dependency

Emotional Video Captioning With Vision-Based Emotion Interpretation Network

Dependency Exploitation: A Unified CNN-RNN Approach for Visual Emotion Recognition

Sentiment-Aware Word Embedding for Emotion Classification