Abstract:Automatic continuous affect recognition from audiovisual cues is arguably one of the most active research areas in machine learning. In addressing this regression problem, the advantages of the models, such as the global-optimisation capability of Support Vector Machine for Regression and the context-sensitive capability of memory-enhanced neural networks, have been frequently explored, but in an isolated way. Motivated to leverage the individual advantages of these techniques, this paper proposes and explores a novel framework, Strength Modelling, where two models are concatenated in a hierarchical framework. In doing this, the strength information of the first model, as represented by its predictions, is joined with the original features, and this expanded feature space is then utilised as the input by the successive model. A major advantage of Strength Modelling, besides its ability to hierarchically explore the strength of different machine learning algorithms, is that it can work together with the conventional feature- and decision-level fusion strategies for multimodal affect recognition. To highlight the effectiveness and robustness of the proposed approach, extensive experiments have been carried out on two time- and value-continuous spontaneous emotion databases (RECOLA and SEMAINE) using audio and video signals. The experimental results indicate that employing Strength Modelling can deliver a significant performance improvement for both arousal and valence in the unimodal and bimodal settings. The results further show that the proposed systems is competitive or outperform the other state-of-the-art approaches, but being with a simple implementation.

Multimodal continuous affect recognition based on LSTM and multiple kernel learning

Multimodal Dimensional Affect Recognition Using Deep Bidirectional Long Short-Term Memory Recurrent Neural Networks

Continuous Affect Recognition with Weakly Supervised Learning

Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network

Deep Neural Network And Switching Kalman Filter Based Continuous Affect Recognition

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

Multimodal Affective Dimension Prediction Using Deep Bidirectional Long Short-Term Memory Recurrent Neural Networks.

Multimodal Multi-task Learning for Dimensional and Continuous Emotion Recognition.

Efficient Spatial Temporal Convolutional Features for Audiovisual Continuous Affect Recognition

Multimodal Continuous Emotion Recognition with Data Augmentation Using Recurrent Neural Networks

Long Short Term Memory Recurrent Neural Network Based Multimodal Dimensional Emotion Recognition

Exploring Multimodal Visual Features For Continuous Affect Recognition

Multi-resolution modulation-filtered cochleagram feature for LSTM-based dimensional emotion recognition from speech

Multi-Modal Multi-Cultural Dimensional Continues Emotion Recognition In Dyadic Interactions

Continuous-Time Audiovisual Fusion with Recurrence vs. Attention for In-The-Wild Affect Recognition

Emotion Recognition Using Multimodal Residual LSTM Network

Multimodal Prediction of Affective Dimensions Via Fusing Multiple Regression Techniques

Analyzing Multimodal Sentiment Via Acoustic- and Visual-LSTM with Channel-Aware Temporal Convolution Network

A multimodal fusion-based deep learning framework combined with local-global contextual TCNs for continuous emotion recognition from videos

Multi-modal Dimensional Emotion Recognition using Recurrent Neural Networks.

Strength Modelling for Real-Worldautomatic Continuous Affect Recognition from Audiovisual Signals