Abstract:The immanent dependencies between audio and visual modalities extracted from video content and the well-established film grammar (i.e., domain knowledge) are important for emotion video recognition and regression. However, these tools have yet to be exploited successfully. Therefore, we propose a multimodal deep regression Bayesian network (MMDRBN) to capture the relationship between audio and visual modalities for emotion video tagging. We then modify the structure of the MMDRBN to incorporate domain knowledge. A regression Bayesian network (RBN) is formed from one latent layer, one visible layer and directed links from the latent layer to the visible layer. RBN is able to fully represent the data, since it captures the dependencies not only among the visible variables but also among the latent variables given visible variables. For the MMDRBN, first, we learn several layers of RBNs using audio and visual modalities, and then stack these RBNs to form two deep networks. A joint representation is obtained from the top layers of the two deep networks, capturing the deep dependencies between audio and visual modalities. We also summarize the main audio and visual elements used by filmmakers to convey emotions and formulate them as semantical meaningful middle-level representation, i.e., attributes. Through these attributes, we construct the knowledge-augmented MMDRBN, which learns a hybrid middle-level video representation using video data and the summarized attributes. Experimental results of both emotion recognition and regression from videos on the LIRIS-ACCEDE database demonstrate that the proposed model can successfully capture the intrinsic connections between audio and visual modalities, and integrate the middle-level representation learning from video data and semantical attributes summarized from film grammar. Thus, it achieves superior performance on emotion video tagging compared to state-of-the-art methods.

Multi‐label Emotion Recognition of Weblog Sentence Based on Bayesian Networks

Emotion Detection in Online Social Networks: A Multilabel Learning Approach

Multi-label Emotion Classification for Tweets in Weibo: Method and Application

Social Media Opinion Summarization Using Emotion Cognition and Convolutional Neural Networks

Sentiment Analysis of Chinese Micro-blogs Based on Emoticons and Emotional Words

Emotion Detection In Online Social Network Based On Multi-Label Learning

A multi-modal and multi-scale emotion-enhanced inference model based on fuzzy recognition

Attention-Based BiLSTM Network with Lexical Feature for Emotion Classification

Text-based emotion classification using emotion cause extraction

Attention-based multi-level image and text sentiment analysis

A Multi-task Neural Approach for Emotion Attribution, Classification and Summarization

Knowledge-Augmented Multimodal Deep Regression Bayesian Networks for Emotion Video Tagging

Emotion Analysis Model of Microblog Comment Text Based on CNN-BiLSTM

A Rule-Based Approach to Emotion Cause Detection for Chinese Micro-Blogs

Emotion Classification in Microblog Texts Using Class Sequential Rules

Sentiment analysis of online public opinion based on CNN-BiLSTM and attention mechanism

Microblog User Emotion Analysis Method Based on Improved Hierarchical Attention Mechanism and BiLSTM

Emotion Mining Research on Micro-Blog

An Empirical Study of Emotion Analysis with Different Distributed Representation Methods for Chinese Microblogs

Multiple Emotional Tagging of Multimedia Data by Exploiting Dependencies among Emotions.

User-based Hierarchical Network of Sina Weibo Emotion Analysis