Abstract:Facial Expression Recognition (FER) can be widely applied to various research areas, such as mental diseases diagnosis and human social/physiological interaction detection. With the emerging advanced technologies in hardware and sensors, FER systems have been developed to support real-world application scenes, instead of laboratory environments. Although the laboratory-controlled FER systems achieve very high accuracy, around 97%, the technical transferring from the laboratory to real-world applications faces a great barrier of very low accuracy, approximately 50%. In this survey, we comprehensively discuss three significant challenges in the unconstrained real-world environments, such as illumination variation, head pose, and subject-dependence, which may not be resolved by only analysing images/videos in the FER system. We focus on those sensors that may provide extra information and help the FER systems to detect emotion in both static images and video sequences. We introduce three categories of sensors that may help improve the accuracy and reliability of an expression recognition system by tackling the challenges mentioned above in pure image/video processing. The first group is detailed-face sensors, which detect a small dynamic change of a face component, such as eye-trackers, which may help differentiate the background noise and the feature of faces. The second is non-visual sensors, such as audio, depth, and EEG sensors, which provide extra information in addition to visual dimension and improve the recognition reliability for example in illumination variation and position shift situation. The last is target-focused sensors, such as infrared thermal sensors, which can facilitate the FER systems to filter useless visual contents and may help resist illumination variation. Also, we discuss the methods of fusing different inputs obtained from multimodal sensors in an emotion system. We comparatively review the most prominent multimodal emotional expression recognition approaches and point out their advantages and limitations. We briefly introduce the benchmark data sets related to FER systems for each category of sensors and extend our survey to the open challenges and issues. Meanwhile, we design a framework of an expression recognition system, which uses multimodal sensor data (provided by the three categories of sensors) to provide complete information about emotions to assist the pure face image/video analysis. We theoretically analyse the feasibility and achievability of our new expression recognition system, especially for the use in the wild environment, and point out the future directions to design an efficient, emotional expression recognition system.

Federated learning inspired privacy sensitive emotion recognition based on multi-modal physiological sensors

AFLEMP: Attention-based Federated Learning for Emotion recognition using Multi-modal Physiological data

Federated learning in Emotion Recognition Systems based on physiological signals for privacy preservation: a review

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

A federated learning method for real-time emotion state classification from multi-modal streaming

EEG Emotion Recognition Based on Federated Learning Framework

Multimodal emotion recognition based on the fusion of vision, EEG, ECG, and EMG signals

A Joint Communication and Federated Learning Framework for Internet of Things Networks

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

Emotion recognition framework using multiple modalities for an effective human–computer interaction

Federated Learning Using Multi-Modal Sensors with Heterogeneous Privacy Sensitivity Levels

Evaluating Ensemble Learning Methods for Multi-Modal Emotion Recognition Using Sensor Data Fusion

Recognition of Emotions Using Multimodal Physiological Signals and an Ensemble Deep Learning Model

Real-time emotional health detection using fine-tuned transfer networks with multimodal fusion

Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems

Automatic Emotion Recognition Using Temporal Multimodal Deep Learning

Emotion recognition based on multiple physiological signals

Novel multimodal emotion detection method using Electroencephalogram and Electrocardiogram signals

Valence-Arousal Model based Emotion Recognition using EEG, peripheral physiological signals and Facial Expression

A Review on Automatic Facial Expression Recognition Systems Assisted by Multimodal Sensor Data