Abstract:Purpose Although numerous signal modalities are available for emotion recognition, audio and visual modalities are the most common and predominant forms for human beings to express their emotional states in daily communication. Therefore, how to achieve automatic and accurate audiovisual emotion recognition is significantly important for developing engaging and empathetic human–computer interaction environment. However, two major challenges exist in the field of audiovisual emotion recognition: (1) how to effectively capture representations of each single modality and eliminate redundant features and (2) how to efficiently integrate information from these two modalities to generate discriminative representations. Design/methodology/approach A novel key-frame extraction-based attention fusion network (KE-AFN) is proposed for audiovisual emotion recognition. KE-AFN attempts to integrate key-frame extraction with multimodal interaction and fusion to enhance audiovisual representations and reduce redundant computation, filling the research gaps of existing approaches. Specifically, the local maximum–based content analysis is designed to extract key-frames from videos for the purpose of eliminating data redundancy. Two modules, including “Multi-head Attention-based Intra-modality Interaction Module” and “Multi-head Attention-based Cross-modality Interaction Module”, are proposed to mine and capture intra- and cross-modality interactions for further reducing data redundancy and producing more powerful multimodal representations. Findings Extensive experiments on two benchmark datasets (i.e. RAVDESS and CMU-MOSEI) demonstrate the effectiveness and rationality of KE-AFN. Specifically, (1) KE-AFN is superior to state-of-the-art baselines for audiovisual emotion recognition. (2) Exploring the supplementary and complementary information of different modalities can provide more emotional clues for better emotion recognition. (3) The proposed key-frame extraction strategy can enhance the performance by more than 2.79 per cent on accuracy. (4) Both exploring intra- and cross-modality interactions and employing attention-based audiovisual fusion can lead to better prediction performance. Originality/value The proposed KE-AFN can support the development of engaging and empathetic human–computer interaction environment.

Integrating gating and learned queries in audiovisual emotion recognition

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

Investigating Multisensory Integration in Emotion Recognition Through Bio-Inspired Computational Models

Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognition

Exploring Emotion Features and Fusion Strategies for Audio-Video Emotion Recognition

Detail-Enhanced Intra- and Inter-modal Interaction for Audio-Visual Emotion Recognition

Analyzing Audiovisual Data for Understanding User's Emotion in Human-Computer Interaction Environment

MGMS: A Modality-General and Modality-Specific Learning Model Using EEG and Facial Expression for Hearing-Impaired Emotion Recognition

An autoencoder-based feature level fusion for speech emotion recognition

Audio-Visual Emotion Recognition Based on Facial Expression and Affective Speech

Multimodal emotion recognition from facial expression and speech based on feature fusion

Audio-Visual Fusion Network Based on Conformer for Multimodal Emotion Recognition

Valence-Arousal Model based Emotion Recognition using EEG, peripheral physiological signals and Facial Expression

Learning Better Representations for Audio-Visual Emotion Recognition with Common Information

Combining cross-modal knowledge transfer and semi-supervised learning for speech emotion recognition

Multimodal interaction enhanced representation learning for video emotion recognition

Context-aware Multimodal Fusion for Emotion Recognition

FV2ES: A Fully End2End Multimodal System for Fast Yet Effective Video Emotion Recognition Inference