Abstract:In recent years, emotion recognition based on electroencephalography (EEG) signals has attracted plenty of attention. Most of the existing works focused on normal or depressed people. Due to the lack of hearing ability, it is difficult for hearing-impaired people to express their emotions through language in their social activities. In this work, we collected the EEG signals of hearing-impaired subjects when they were watching six kinds of emotional video clips (happiness, inspiration, neutral, anger, fear, and sadness) for emotion recognition. The biharmonic spline interpolation method was utilized to convert the traditional frequency domain features, Differential Entropy (DE), Power Spectral Density (PSD), and Wavelet Entropy (WE) into the spatial domain. The patch embedding (PE) method was used to segment the feature map into the same patch to obtain the differences in the distribution of emotional information among brain regions. For feature classification, a compact residual network with Depthwise convolution (DC) and Pointwise convolution (PC) is proposed to separate spatial and channel mixing dimensions to better extract information between channels. Dependent subject experiments based on 70% training sets and 30% testing sets were performed. The results showed that the average classification accuracies by PE (DE), PE (PSD), and PE (WE) were 91.75%, 85.53%, and 75.68%, respectively which were improved by 11.77%, 23.54%, and 16.61% compared with DE, PSD, and WE. Moreover, the comparison experiments were carried out on the SEED and DEAP datasets with PE (DE), which achieved average accuracies of 90.04% (positive, neutral, and negative) and 88.75% (high valence and low valence). By exploring the emotional brain regions, we found that the frontal, parietal, and temporal lobes of hearing-impaired people were associated with emotional activity compared to normal people whose main emotional brain area was the frontal lobe.

MGMS: A Modality-General and Modality-Specific Learning Model Using EEG and Facial Expression for Hearing-Impaired Emotion Recognition

Image-Evoked Emotion Recognition for Hearing-Impaired Subjects with EEG Signals

Emotion Recognition of Subjects With Hearing Impairment Based on Fusion of Facial Expression and EEG Topographic Map

Investigating of Deaf Emotion Cognition Pattern by EEG and Facial Expression Combination

EEG-Based Emotion Recognition for Hearing Impaired and Normal Individuals With Residual Feature Pyramids Network Based on Time–Frequency–Spatial Features

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

Facial Expression and EEG Fusion for Investigating Continuous Emotions of Deaf Subjects

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Valence-Arousal Model based Emotion Recognition using EEG, peripheral physiological signals and Facial Expression

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

Emotion recognition with residual network driven by spatial-frequency characteristics of EEG recorded from hearing-impaired adults in response to video clips

Multimodal Emotion Recognition by Extracting Common and Modality-Specific Information.

Multi-modal emotion recognition using EEG and speech signals

Multimodal Emotion Recognition based on the Fusion of EEG Signals and Eye Movement Data

Emotion Recognition Using Cross-Modal Attention from Eeg and Facial Expression

An Attention-Based Multi-Domain Bi-Hemisphere Discrepancy Feature Fusion Model for EEG Emotion Recognition

Multi-Modal Emotion Recognition Based On deep Learning Of EEG And Audio Signals

Multimodal emotion recognition model via hybrid model with improved feature level fusion on facial and EEG feature set

EEG-SCMM: Soft Contrastive Masked Modeling for Cross-Corpus EEG-Based Emotion Recognition

Emotion Analysis: Bimodal Fusion of Facial Expressions and EEG

Multimodal Multi-View Spectral-Spatial-Temporal Masked Autoencoder for Self-Supervised Emotion Recognition