Abstract:Multimodal emotion recognition of videos of human oration, commonly called opinion videos, has a wide scope of applications across all domains. Here, the speakers express their views or opinions about various topics. This field is being researched by many with the aim of introducing accurate and efficient architectures for the same. This study also carries the same objective while exploring novel concepts in the field of emotion recognition. The proposed framework uses cross-dataset training and testing, so that the resultant architecture and models are unrestricted by the domain of input. It uses benchmark datasets and ensemble learning to make sure that even if the individual models are slightly biased, they can be countered by the learnings of the other models. Therefore, to achieve this objective with the mentioned novelties, three benchmark datasets, ISEAR, RAVDESS, and FER-2013, are used to train independent models for each of the three modalities of text, audio, and images. Another dataset is used in addition to the ISEAR dataset to train the text model. They are then combined and tested on the benchmark multimodal dataset of CMU-MOSEI. For the text analysis model, ELMo embedding and RNN are used, for audio, a simple DNN is used and for image emotion recognition, a 2D CNN is used after pre-processing. They are aggregated using the stacking technique for the final result. The complete architecture can be used as a partially pre-trained algorithm for the prediction of individual modalities, and partially trainable for stacking the results to get efficient emotion prediction based on input quality. The accuracy obtained on the CMU-MOSEI data set is 86.60% and the F1-score for the same is 0.84.

All-in-One: Emotion, Sentiment and Intensity Prediction Using a Multi-Task Ensemble Framework

Emotion and sentiment analysis for intelligent customer service conversation using a multi-task ensemble framework

Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

A Deep Multi-task Contextual Attention Framework for Multi-modal Affect Analysis

Multi-modal Sentiment and Emotion Joint Analysis with a Deep Attentive Multi-task Learning Model

Multi-Task Learning Framework for Emotion Recognition In-the-Wild

Towards a One-stop Solution to Both Aspect Extraction and Sentiment Analysis Tasks with Neural Multi-task Learning.

A Multitask Multimodal Ensemble Model for Sentiment- and Emotion-Aided Tweet Act Classification

Deep-Sentiment: Sentiment Analysis Using Ensemble of CNN and Bi-LSTM Models

Hybrid Multi-Task Learning for End-To-End Multimodal Emotion Recognition

A Multi-Task Learning Framework for Emotion Recognition Using 2D Continuous Space.

DJMF: A Discriminative Joint Multi-Task Framework for Multimodal Sentiment Analysis Based on Intra- and Inter-Task Dynamics

A customizable framework for multimodal emotion recognition using ensemble of deep neural network models

Multi-model Ensemble Learning Method for Human Expression Recognition

Multi-Task Learning for Jointly Detecting Depression and Emotion.

A Multitask Learning Framework for Abuse Detection and Emotion Classification

A Novel Multi-Feature Joint Learning Ensemble Framework for Multi-Label Facial Expression Recognition

A Multi-task Neural Approach for Emotion Attribution, Classification and Summarization