Abstract:Human facial expressions are regarded as a vital indicator of one’s emotion and intention, and even reveal the state of health and wellbeing. Emotional states have been associated with information processing within and between subcortical and cortical areas of the brain, including the amygdala and prefrontal cortex. In this study, we evaluated the relationship between spontaneous human facial affective expressions and multi-modal brain activity measured via non-invasive and wearable sensors: functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) signals. The affective states of twelve male participants detected via fNIRS, EEG, and spontaneous facial expressions were investigated in response to both image-content stimuli and video-content stimuli. We propose a method to jointly evaluate fNIRS and EEG signals for affective state detection (emotional valence as positive or negative). Experimental results reveal a strong correlation between spontaneous facial affective expressions and the perceived emotional valence. Moreover, the affective states were estimated by the fNIRS, EEG, and fNIRS + EEG brain activity measurements. We show that the proposed EEG + fNIRS hybrid method outperforms fNIRS-only and EEG-only approaches. Our findings indicate that the dynamic (video-content based) stimuli triggers a larger affective response than the static (image-content based) stimuli. These findings also suggest joint utilization of facial expression and wearable neuroimaging, fNIRS, and EEG, for improved emotional analysis and affective brain–computer interface applications.

Multimodal Affect Analysis for Product Feedback Assessment

Multimodal Affect Recognition using Kinect

Intelligent emotion recognition in product design using multimodal physiological signals and machine learning

A Multimodal Emotion Recognition System: Integrating Facial Expressions, Body Movement, Speech, and Spoken Language

Digital Integration and Automated Assessment of Eye-Tracking and Emotional Response Data Using the BioSensory App to Maximize Packaging Label Analysis

Multimodal Emotion Recognition by Combining Physiological Signals and Facial Expressions: a Preliminary Study.

A Multimodal Emotion Sensing Platform for Building Emotion-Aware Applications

ASSESSMENT OF FACIAL EXPRESSIONS IN PRODUCT APPRECIATION

Multimodal Characterization of Emotion within Multimedia Space

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Multi-modal Approach for Affective Computing

Critical Analysis on Multimodal Emotion Recognition in Meeting the Requirements for Next Generation Human Computer Interactions

Three-Dimensional, Kinematic, Human Behavioral Pattern-Based Features for Multimodal Emotion Recognition

Modeling And Application Of Multimodal Affective User Interface With Multimedia

Online Feedback System for Public Speakers

IMPACT OF VISUAL MODALITIES IN MULTIMODAL PERSONALITY AND AFFECTIVE COMPUTING

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

Multimodal Prediction of Affective Dimensions and Depression in Human-Computer Interactions

Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages

Multimodal human attention detection for reading from facial expression, eye gaze, and mouse dynamics