Abstract:Purpose Although numerous signal modalities are available for emotion recognition, audio and visual modalities are the most common and predominant forms for human beings to express their emotional states in daily communication. Therefore, how to achieve automatic and accurate audiovisual emotion recognition is significantly important for developing engaging and empathetic human–computer interaction environment. However, two major challenges exist in the field of audiovisual emotion recognition: (1) how to effectively capture representations of each single modality and eliminate redundant features and (2) how to efficiently integrate information from these two modalities to generate discriminative representations. Design/methodology/approach A novel key-frame extraction-based attention fusion network (KE-AFN) is proposed for audiovisual emotion recognition. KE-AFN attempts to integrate key-frame extraction with multimodal interaction and fusion to enhance audiovisual representations and reduce redundant computation, filling the research gaps of existing approaches. Specifically, the local maximum–based content analysis is designed to extract key-frames from videos for the purpose of eliminating data redundancy. Two modules, including “Multi-head Attention-based Intra-modality Interaction Module” and “Multi-head Attention-based Cross-modality Interaction Module”, are proposed to mine and capture intra- and cross-modality interactions for further reducing data redundancy and producing more powerful multimodal representations. Findings Extensive experiments on two benchmark datasets (i.e. RAVDESS and CMU-MOSEI) demonstrate the effectiveness and rationality of KE-AFN. Specifically, (1) KE-AFN is superior to state-of-the-art baselines for audiovisual emotion recognition. (2) Exploring the supplementary and complementary information of different modalities can provide more emotional clues for better emotion recognition. (3) The proposed key-frame extraction strategy can enhance the performance by more than 2.79 per cent on accuracy. (4) Both exploring intra- and cross-modality interactions and employing attention-based audiovisual fusion can lead to better prediction performance. Originality/value The proposed KE-AFN can support the development of engaging and empathetic human–computer interaction environment.

An efficient model-level fusion approach for continuous affect recognition from audiovisual signals

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition

Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognition

A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition

Audio-Visual Fusion Network Based on Conformer for Multimodal Emotion Recognition

Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention

Multimodal Transformer Fusion for Continuous Emotion Recognition

Multimodal emotion recognition from facial expression and speech based on feature fusion

An Improved Multimodal Dimension Emotion Recognition Based on Different Fusion Methods

A multimodal fusion-based deep learning framework combined with local-global contextual TCNs for continuous emotion recognition from videos

Transformer-based Multimodal Information Fusion for Facial Expression Analysis

Detail-Enhanced Intra- and Inter-modal Interaction for Audio-Visual Emotion Recognition

Transformer Encoder With Multi-Modal Multi-Head Attention for Continuous Affect Recognition

3-D Facial Expression Recognition via Attention-Based Multichannel Data Fusion Network

A Multimodal Sentiment Analysis Approach Based on a Joint Chained Interactive Attention Mechanism

Analyzing Audiovisual Data for Understanding User's Emotion in Human-Computer Interaction Environment

Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusion

Affective Video Classification Based on Spatio-temporal Feature Fusion

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features