Multi-view Emotional Expressions Dataset

Mingming Zhang,Yangen Zhou,Xiaojun Xu,Ren Ziwei,Yihan Zhang,Shenglan Liu,Wenbo Luo

DOI: https://doi.org/10.5281/zenodo.7779985

2023-01-01

Abstract:Multi-view emotional expressions dataset (MEED) using 2D pose estimation.

What problem does this paper attempt to address?

Multi-view Emotional Expressions Dataset Using 2D Pose Estimation.

Mingming Zhang,Yanan Zhou,Xinye Xu,Ziwei Ren,Yihan Zhang,Shenglan Liu,Wenbo Luo

DOI: https://doi.org/10.1038/s41597-023-02551-y

2023-01-01

Scientific Data

Abstract:Human body expressions convey emotional shifts and intentions of action and, in some cases, are even more effective than other emotion models. Despite many datasets of body expressions incorporating motion capture available, there is a lack of more widely distributed datasets regarding naturalized body expressions based on the 2D video. In this paper, therefore, we report the multi-view emotional expressions dataset (MEED) using 2D pose estimation. Twenty-two actors presented six emotional (anger, disgust, fear, happiness, sadness, surprise) and neutral body movements from three viewpoints (left, front, right). A total of 4102 videos were captured. The MEED consists of the corresponding pose estimation results (i.e., 397,809 PNG files and 397,809 JSON files). The size of MEED exceeds 150 GB. We believe this dataset will benefit the research in various fields, including affective computing, human-computer interaction, social neuroscience, and psychiatry.
4DME: A Spontaneous 4D Micro-Expression Dataset with Multimodalities

Xiaobai Li,Shiyang Cheng,Yante Li,Muzammil Behzad,Jie Shen,Stefanos Zafeiriou,Maja Pantic,Guoying Zhao

DOI: https://doi.org/10.1109/taffc.2022.3182342

IF: 13.99

2023-01-01

IEEE Transactions on Affective Computing

Abstract:Micro-expressions (ME) are a special form of facial expressions which may occur when people try to hide their true feelings for some reasons. MEs are important clues to reveal people's true feelings, but are difficult or impossible to be captured by ordinary persons with naked-eyes as they are very short and subtle. It is expected that robust computer vision methods can be developed to automatically analyze MEs which requires lots of ME data. The current ME datasets are insufficient, and mostly contain only one single form of 2D color videos. Researches on 4D data of ordinary facial expressions have prospered, but so far no 4D data is available in ME study. In the current study, we introduce the 4DME dataset: a new spontaneous ME dataset which includes 4D data along with three other video modalities. Both micro- and macro-expression clips are labeled out in 4DME, and 22 AU labels and five categories of emotion labels are annotated. Experiments are carried out using three 2D-based methods and one 4D-based method to provide baseline results. The results indicate that the 4D data can potentially benefit ME recognition. The 4DME dataset could be used for developing 4D-based approaches, or exploring fusion of multiple video sources (e.g., texture and depth) for the task of ME analysis in future. Besides, we also emphasize the importance of forming a clear and unified criteria of ME annotation for future ME data collection studies. Several key questions related with ME annotation are listed and discussed in depth, especially about the relationship between AUs and ME emotion categories. A preliminary AU-Emo mapping table is proposed with justified explanations and supportive experimental results. Several unsolved issues are also summarized for future work.
M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database

Jinming Zhao,Tenggan Zhang,Jingwen Hu,Yuchen Liu,Qin Jin,Xinchao Wang,Haizhou Li

DOI: https://doi.org/10.48550/arXiv.2205.10237

2022-05-09

Abstract:The emotional state of a speaker can be influenced by many different factors in dialogues, such as dialogue scene, dialogue topic, and interlocutor stimulus. The currently available data resources to support such multimodal affective analysis in dialogues are however limited in scale and diversity. In this work, we propose a Multi-modal Multi-scene Multi-label Emotional Dialogue dataset, M3ED, which contains 990 dyadic emotional dialogues from 56 different TV series, a total of 9,082 turns and 24,449 utterances. M3 ED is annotated with 7 emotion categories (happy, surprise, sad, disgust, anger, fear, and neutral) at utterance level, and encompasses acoustic, visual, and textual modalities. To the best of our knowledge, M3ED is the first multimodal emotional dialogue dataset in Chinese. It is valuable for cross-culture emotion analysis and recognition. We apply several state-of-the-art methods on the M3ED dataset to verify the validity and quality of the dataset. We also propose a general Multimodal Dialogue-aware Interaction framework, MDI, to model the dialogue context for emotion recognition, which achieves comparable performance to the state-of-the-art methods on the M3ED. The full dataset and codes are available.

Computation and Language,Artificial Intelligence
MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation

Kaisiyuan Wang,Qianyi Wu,Linsen Song,Zhuoqian Yang,Wayne Wu,Chen Qian,Ran He,Yu Qiao,Chen Change Loy

DOI: https://doi.org/10.1007/978-3-030-58589-1_42

2020-01-01

Abstract:The synthesis of natural emotional reactions is an essential criterion in vivid talking-face video generation. This criterion is nevertheless seldom taken into consideration in previous works due to the absence of a large-scale, high-quality emotional audio-visual dataset. To address this issue, we build the Multi-view Emotional Audio-visual Dataset (MEAD), a talking-face video corpus featuring 60 actors and actresses talking with eight different emotions at three different intensity levels. High-quality audio-visual clips are captured at seven different view angles in a strictly-controlled environment. Together with the dataset, we release an emotional talking-face generation baseline that enables the manipulation of both emotion and its intensity. Our dataset could benefit a number of different research fields including conditional generation, cross-modal understanding and expression recognition. Code, model and data are publicly available on our project page ‡$$^{\ddagger }$$‡$$^{\ddagger }$$https://wywu.github.io/projects/MEAD/MEAD.html.
MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition

Tengfei Song,Wenming Zheng,Cheng Lu,Yuan Zong,Xilei Zhang,Zhen Cui

DOI: https://doi.org/10.1109/access.2019.2891579

IF: 3.9

2019-01-01

IEEE Access

Abstract:To explore human emotions, in this paper, we design and build a multi-modal physiological emotion database, which collects four modal physiological signals, i.e., electroencephalogram (EEG), galvanic skin response, respiration, and electrocardiogram (ECG). To alleviate the influence of culture dependent elicitation materials and evoke desired human emotions, we specifically collect an emotion elicitation material database selected from more than 1500 video clips. By the considerable amount of strict man-made labeling, we elaborately choose 28 videos as standardized elicitation samples, which are assessed by psychological methods. The physiological signals of participants were synchronously recorded when they watched these standardized video clips that described six discrete emotions and neutral emotion. With three types of classification protocols, different feature extraction methods and classifiers (support vector machine and k-NearestNeighbor) were used to recognize the physiological responses of different emotions, which presented the baseline results. Simultaneously, we present a novel attention-long short-term memory (A-LSTM), which strengthens the effectiveness of useful sequences to extract more discriminative features. In addition, correlations between the EEG signals and the participants' ratings are investigated. The database has been made publicly available to encourage other researchers to use it to evaluate their own emotion estimation methods.

computer science, information systems,telecommunications,engineering, electrical & electronic
HEU Emotion: A Large-scale Database for Multi-modal Emotion Recognition in the Wild

Jing Chen,Chenhui Wang,Kejun Wang,Chaoqun Yin,Cong Zhao,Tao Xu,Xinyi Zhang,Ziqiang Huang,Meichen Liu,Tao Yang

DOI: https://doi.org/10.1007/s00521-020-05616-w

2020-07-24

Abstract:The study of affective computing in the wild setting is underpinned by databases. Existing multimodal emotion databases in the real-world conditions are few and small, with a limited number of subjects and expressed in a single language. To meet this requirement, we collected, annotated, and prepared to release a new natural state video database (called HEU Emotion). HEU Emotion contains a total of 19,004 video clips, which is divided into two parts according to the data source. The first part contains videos downloaded from Tumblr, Google, and Giphy, including 10 emotions and two modalities (facial expression and body posture). The second part includes corpus taken manually from movies, TV series, and variety shows, consisting of 10 emotions and three modalities (facial expression, body posture, and emotional speech). HEU Emotion is by far the most extensive multi-modal emotional database with 9,951 subjects. In order to provide a benchmark for emotion recognition, we used many conventional machine learning and deep learning methods to evaluate HEU Emotion. We proposed a Multi-modal Attention module to fuse multi-modal features adaptively. After multi-modal fusion, the recognition accuracies for the two parts increased by 2.19% and 4.01% respectively over those of single-modal facial expression recognition.

Computer Vision and Pattern Recognition,Machine Learning
An EEG-Based Multi-Modal Emotion Database with Both Posed and Authentic Facial Actions for Emotion Analysis

Xiaotian Li,Xiang Zhang,Huiyuan Yang,Wenna Duan,Weiying Dai,Lijun Yin

DOI: https://doi.org/10.48550/arXiv.2203.15829

2022-03-29

Computer Vision and Pattern Recognition

Abstract:Emotion is an experience associated with a particular pattern of physiological activity along with different physiological, behavioral and cognitive changes. One behavioral change is facial expression, which has been studied extensively over the past few decades. Facial behavior varies with a person's emotion according to differences in terms of culture, personality, age, context, and environment. In recent years, physiological activities have been used to study emotional responses. A typical signal is the electroencephalogram (EEG), which measures brain activity. Most of existing EEG-based emotion analysis has overlooked the role of facial expression changes. There exits little research on the relationship between facial behavior and brain signals due to the lack of dataset measuring both EEG and facial action signals simultaneously. To address this problem, we propose to develop a new database by collecting facial expressions, action units, and EEGs simultaneously. We recorded the EEGs and face videos of both posed facial actions and spontaneous expressions from 29 participants with different ages, genders, ethnic backgrounds. Differing from existing approaches, we designed a protocol to capture the EEG signals by evoking participants' individual action units explicitly. We also investigated the relation between the EEG signals and facial action units. As a baseline, the database has been evaluated through the experiments on both posed and spontaneous emotion recognition with images alone, EEG alone, and EEG fused with images, respectively. The database will be released to the research community to advance the state of the art for automatic emotion recognition.
Multi-View Common Space Learning For Emotion Recognition In The Wild

Jianlong Wu,Zhouchen Lin,Hongbin Zha

DOI: https://doi.org/10.1145/2993148.2997631

2016-01-01

Abstract:It is a very challenging task to recognize emotion in the wild. Recently, combining information from various views or modalities has attracted more attention. Cross modality features and features extracted by different methods are regarded as multi-view information of the sample. In this paper, we propose a method to analyse multi-view features of emotion samples and automatically recognize the expression as part of the fourth Emotion Recognition in the Wild Challenge (EmotiW 2016). In our method, we first extract multi-view features such as BoF, CNN, LBP-TOP and audio features for each expression sample. Then we learn the corresponding projection matrices to map multi-view features into a common subspace. In the meantime, we impose l(2,1)-norm penalties on projection matrices for feature selection. We apply both this method and PLSR to emotion recognition. We conduct experiments on both AFEW and HAPPEI datasets, and achieve superior performance. The best recognition accuracy of our method is 55.31% on the AFEW dataset for video based emotion recognition in the wild. The minimum RMSE for group happiness intensity recognition is 0.9525 on HAPPEI dataset. Both of them are much better than that of the challenge baseline.
Mfermeta++ : Robust Multiview Facial Expression Recognition Based on Metahuman and Metalearning

Shuo Jiang,Jiahang Liu,Yanmin Zhou,Zhipeng Wang,Bin He

DOI: https://doi.org/10.1002/aisy.202300210

IF: 7.298

2023-01-01

Advanced Intelligent Systems

Abstract:Facial pose variation presents a significant challenge to facial expression recognition (FER) in real‐world applications. Significant bottlenecks exist in the field of multiview facial expression recognition (MFER) including a lack of high‐quality MFER datasets, and the limited model robustness in real‐world MFER scenarios. Therefore, this article first introduces a metahuman‐based MFER dataset (MMED), which effectively addresses the issues of insufficient quantity and quality in existing datasets. Second, a conditional cascade VGG (ccVGG) model is proposed, which can adaptively adjust expression feature extraction based on the input image's pose information. Finally, a hybrid training and few‐shot learning strategy are proposed that integrates our MMED dataset with a real‐world dataset and quickly deploys it in real‐world application scenarios using the proposed Meta‐Dist few‐shot learning method. Experiments on the Karolinska Directed Emotional Face (KDEF) dataset demonstrate that the proposed model exhibits improved robustness in multiview application scenarios and achieves a recognition accuracy improvement of 28.68% relative to the baseline. It demonstrates that the proposed MMED dataset can effectively improve the training efficiency of MFER models and facilitate easy deployment in real‐world applications. This work provides a reliable dataset for the MFER studies and paves the way for robust FER in any view for real‐world deployment.
Emo3D: Metric and Benchmarking Dataset for 3D Facial Expression Generation from Emotion Description

Mahshid Dehghani,Amirahmad Shafiee,Ali Shafiei,Neda Fallah,Farahmand Alizadeh,Mohammad Mehdi Gholinejad,Hamid Behroozi,Jafar Habibi,Ehsaneddin Asgari

2024-10-03

Abstract:Existing 3D facial emotion modeling have been constrained by limited emotion classes and insufficient datasets. This paper introduces "Emo3D", an extensive "Text-Image-Expression dataset" spanning a wide spectrum of human emotions, each paired with images and 3D blendshapes. Leveraging Large Language Models (LLMs), we generate a diverse array of textual descriptions, facilitating the capture of a broad spectrum of emotional expressions. Using this unique dataset, we conduct a comprehensive evaluation of language-based models' fine-tuning and vision-language models like Contranstive Language Image Pretraining (CLIP) for 3D facial expression synthesis. We also introduce a new evaluation metric for this task to more directly measure the conveyed emotion. Our new evaluation metric, Emo3D, demonstrates its superiority over Mean Squared Error (MSE) metrics in assessing visual-text alignment and semantic richness in 3D facial expressions associated with human emotions. "Emo3D" has great applications in animation design, virtual reality, and emotional human-computer interaction.

Computer Vision and Pattern Recognition,Computation and Language,Graphics
EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes

Jingyuan Yang,Qirui Huang,Tingting Ding,Dani Lischinski,Daniel Cohen-Or,Hui Huang,Qiruin Huang

DOI: https://doi.org/10.48550/arXiv.2307.07961

2023-07-16

Computer Vision and Pattern Recognition

Abstract:Visual Emotion Analysis (VEA) aims at predicting people's emotional responses to visual stimuli. This is a promising, yet challenging, task in affective computing, which has drawn increasing attention in recent years. Most of the existing work in this area focuses on feature design, while little attention has been paid to dataset construction. In this work, we introduce EmoSet, the first large-scale visual emotion dataset annotated with rich attributes, which is superior to existing datasets in four aspects: scale, annotation richness, diversity, and data balance. EmoSet comprises 3.3 million images in total, with 118,102 of these images carefully labeled by human annotators, making it five times larger than the largest existing dataset. EmoSet includes images from social networks, as well as artistic images, and it is well balanced between different emotion categories. Motivated by psychological studies, in addition to emotion category, each image is also annotated with a set of describable emotion attributes: brightness, colorfulness, scene type, object class, facial expression, and human action, which can help understand visual emotions in a precise and interpretable way. The relevance of these emotion attributes is validated by analyzing the correlations between them and visual emotion, as well as by designing an attribute module to help visual emotion recognition. We believe EmoSet will bring some key insights and encourage further research in visual emotion analysis and understanding. The data and code will be released after the publication of this work.
Construction and Validation of the Dalian Emotional Movement Open-Source Set (DEMOS)

Mingming Zhang,Lu Yu,Keye Zhang,Bixuan Du,Bin Zhan,Shuxin Jia,Shaohua Chen,Fengxu Han,Yiwen Li,Shuaicheng Liu,Xi Yi,Shenglan Liu,Wenbo Luo

DOI: https://doi.org/10.3758/s13428-022-01887-4

IF: 5.953

2022-01-01

Behavior Research Methods

Abstract:Human body movements are important for emotion recognition and social communication and have received extensive attention from researchers. In this field, emotional biological motion stimuli, as depicted by point-light displays, are widely used. However, the number of stimuli in the existing material library is small, and there is a lack of standardized indicators, which subsequently limits experimental design and conduction. Therefore, based on our prior kinematic dataset, we constructed the Dalian Emotional Movement Open-source Set (DEMOS) using computational modeling. The DEMOS has three views (i.e., frontal 0°, left 45°, and left 90°) and in total comprises 2664 high-quality videos of emotional biological motion, each displaying happiness, sadness, anger, fear, disgust, and neutral. All stimuli were validated in terms of recognition accuracy, emotional intensity, and subjective movement. The objective movement for each expression was also calculated. The DEMOS can be downloaded for free from https://osf.io/83fst/ . To our knowledge, this is the largest multi-view emotional biological motion set based on the whole body. The DEMOS can be applied in many fields, including affective computing, social cognition, and psychiatry.
HEU Emotion: a Large-Scale Database for Multimodal Emotion Recognition in the Wild

Chen Jing,Wang Chenhui,Wang Kejun,Yin Chaoqun,Zhao Cong,Xu Tao,Zhang Xinyi,Huang Ziqiang,Liu Meichen,Yang Tao

DOI: https://doi.org/10.1007/s00521-020-05616-w

2021-01-01

Neural Computing and Applications

Abstract:The study of affective computing in the wild setting is underpinned by databases. Existing multimodal emotion databases in the real-world conditions are few and small, with a limited number of subjects and expressed in a single language. To meet this requirement, we collected, annotated, and prepared to release a new natural state video database (called HEU Emotion). HEU Emotion contains a total of 19,004 video clips, which is divided into two parts according to the data source. The first part contains videos downloaded from Tumblr, Google, and Giphy, including 10 emotions and two modalities (facial expression and body posture). The second part includes corpus taken manually from movies, TV series, and variety shows, consisting of 10 emotions and three modalities (facial expression, body posture, and emotional speech). HEU Emotion is by far the most extensive multimodal emotional database with 9951 subjects. In order to provide a benchmark for emotion recognition, we used many conventional machine learning and deep learning methods to evaluate HEU Emotion. We proposed a multimodal attention module to fuse multimodal features adaptively. After multimodal fusion, the recognition accuracies for the two parts increased by 2.19% and 4.01%, respectively, over those of single-modal facial expression recognition.
More is Better: A Database for Spontaneous Micro-Expression with High Frame Rates

Sirui Zhao,Huaying Tang,Xinglong Mao,Shifeng Liu,Hanqing Tao,Hao Wang,Tong Xu,Enhong Chen

DOI: https://doi.org/10.48550/arXiv.2301.00985

2023-01-03

Abstract:As one of the most important psychic stress reactions, micro-expressions (MEs), are spontaneous and transient facial expressions that can reveal the genuine emotions of human beings. Thus, recognizing MEs (MER) automatically is becoming increasingly crucial in the field of affective computing, and provides essential technical support in lie detection, psychological analysis and other areas. However, the lack of abundant ME data seriously restricts the development of cutting-edge data-driven MER models. Despite the recent efforts of several spontaneous ME datasets to alleviate this problem, it is still a tiny amount of work. To solve the problem of ME data hunger, we construct a dynamic spontaneous ME dataset with the largest current ME data scale, called DFME (Dynamic Facial Micro-expressions), which includes 7,526 well-labeled ME videos induced by 671 participants and annotated by more than 20 annotators throughout three years. Afterwards, we adopt four classical spatiotemporal feature learning models on DFME to perform MER experiments to objectively verify the validity of DFME dataset. In addition, we explore different solutions to the class imbalance and key-frame sequence sampling problems in dynamic MER respectively on DFME, so as to provide a valuable reference for future research. The comprehensive experimental results show that our DFME dataset can facilitate the research of automatic MER, and provide a new benchmark for MER. DFME will be published via <a class="link-external link-https" href="https://mea-lab-421.github.io" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
MEDA-CBLSTM: Data Acquisition, Processing and Emotion Recognition Based on Multi-Layer EDA

Yuchen He,Lijiang Chen,Qi Zhao,Zhibo Hong,Yu Chen

DOI: https://doi.org/10.1109/AIHCIR61661.2023.00066

2023-01-01

Abstract:Using wearable devices to extract physiological information to identify and classify human emotion is a common research problem in human-computer interaction. From a computational point of view, emotion is difficult to quantify, estimate, and understand. However, indications of human cognitive and emotional related processes affect a variety of derived changes in many human psychological signals. These signals can help to build a quantitative model of emotion. We design and build a set of wearable hardware to acquisit, extract and process Multi-layer Electrodermal activity(MEDA) physiological signal data,and bulid neural network models based on the MEDA data. We build the dataset useing videos that can trigger happiness, sadness, anger, etc. For the annotated and sorted data set, we build a multilayer neural network model to perform feature extraction, mapping, and recognition from MEDA to emotions. Ablation experiments and results comparison are carried out for each model. Experiments show the hybrid model: end-to-end convolutional neural network with bidirectional long short-term memory network(MEDA-CBLSTM) achieves the best result. The model trained by the train set of testers achieves an average accuracy of 92.31% in their corresponding test set, but the accuracy reduces when it is tested on the other test sets. Therefore, we get the conclusion:the EDA emotion data has the characteristics of highly subject dependence.
Beihang University Facial Expression Database and Multiple Facial Expression Recognition

Yuli Xue,Xia Mao,Fan Zhang

DOI: https://doi.org/10.1109/ICMLC.2006.258460

2006-01-01

Abstract:Over the last decade, significant effort has been made in developing methods of facial expression analysis. Because of the limited data sets of facial expressions, comprehensive emotional facial expression recognition that is essential in the areas such as affective computing and human-machine interaction has not been well studied. This paper provides the Beihang University facial expression database, which includes 25 facial expressions (18 pure facial expressions, 3 mixed facial expressions and 4 complex facial expressions) and provides data of many unique emotional facial expressions not appeared in other face databases. The questionnaire investigation of videos of 18 pure facial expressions shows the effectiveness of the data. Facial expression recognition experiment on 9 pure facial expressions demonstrates the feasibility of multiple facial expression recognition
SEED-VII: A Multimodal Dataset of Six Basic Emotions with Continuous Labels for Emotion Recognition

Wei-Bang Jiang,Xuan-Hao Liu,Wei-Long Zheng,Bao-Liang Lu

DOI: https://doi.org/10.1109/taffc.2024.3485057

IF: 13.99

2024-01-01

IEEE Transactions on Affective Computing

Abstract:Recognizing emotions from physiological signals is a topic that has garnered widespread interest, and research continues to develop novel techniques for perceiving emotions. However, the emergence of deep learning has highlighted the need for comprehensive and high-quality emotional datasets that enable the accurate decoding of human emotions. To systematically explore human emotions, we develop a multimodal dataset consisting of six basic (happiness, sadness, fear, disgust, surprise, and anger) emotions and the neutral emotion, named SEED-VII. This multimodal dataset includes electroencephalography (EEG) and eye movement signals. The seven emotions in SEED-VII are elicited by 80 different videos and fully investigated with continuous labels that indicate the intensity levels of the corresponding emotions. Additionally, we propose a novel Multimodal Adaptive Emotion Transformer (MAET), that can flexibly process both unimodal and multimodal inputs. Adversarial training is utilized in the MAET to mitigate subject discrepancies, which enhances domain generalization. Our extensive experiments, encompassing both subject-dependent and cross-subject conditions, demonstrate the superior performance of the MAET in terms of handling various inputs. Continuous labels are used to filter the data with high emotional intensity, and this strategy is proven to be effective for attaining improved emotion recognition performance. Furthermore, complementary properties between the EEG signals and eye movements and stable neural patterns of the seven emotions are observed.
MEDIC: A Multimodal Empathy Dataset in Counseling

Zhou'an_Zhu,Xin Li,Jicai Pan,Yufei Xiao,Yanan Chang,Feiyi Zheng,Shangfei Wang

DOI: https://doi.org/10.48550/arXiv.2305.02842

2023-05-04

Computer Vision and Pattern Recognition

Abstract:Although empathic interaction between counselor and client is fundamental to success in the psychotherapeutic process, there are currently few datasets to aid a computational approach to empathy understanding. In this paper, we construct a multimodal empathy dataset collected from face-to-face psychological counseling sessions. The dataset consists of 771 video clips. We also propose three labels (i.e., expression of experience, emotional reaction, and cognitive reaction) to describe the degree of empathy between counselors and their clients. Expression of experience describes whether the client has expressed experiences that can trigger empathy, and emotional and cognitive reactions indicate the counselor's empathic reactions. As an elementary assessment of the usability of the constructed multimodal empathy dataset, an interrater reliability analysis of annotators' subjective evaluations for video clips is conducted using the intraclass correlation coefficient and Fleiss' Kappa. Results prove that our data annotation is reliable. Furthermore, we conduct empathy prediction using three typical methods, including the tensor fusion network, the sentimental words aware fusion network, and a simple concatenation model. The experimental results show that empathy can be well predicted on our dataset. Our dataset is available for research purposes.
Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels

Wei-Bang Jiang,Xuan-Hao Liu,Wei-Long Zheng,Bao-Liang Lu

DOI: https://doi.org/10.1145/3581783.3613797

2023-01-01

Abstract:Emotion recognition from physiological signals is a topic of widespread interest, and researchers continue to develop novel techniques for perceiving emotions. However, the emergence of deep learning has highlighted the need for high-quality emotional datasets to accurately decode human emotions. In this study, we present a novel multimodal emotion dataset that incorporates electroencephalography (EEG) and eye movement signals to systematically explore human emotions. Seven basic emotions (happy, sad, fear, disgust, surprise, anger, and neutral) are elicited by a large number of 80 videos and fully investigated with continuous labels that indicate the intensity of the corresponding emotions. Additionally, we propose a novel Multimodal Adaptive Emotion Transformer (MAET), that can flexibly process both unimodal and multimodal inputs. Adversarial training is utilized in MAET to mitigate subject discrepancy, which enhances domain generalization. Our extensive experiments, encompassing both subject-dependent and cross-subject conditions, demonstrate MAET's superior performance in handling various inputs. The filtering of data for high emotional evocation using continuous labels proved to be effective in the experiments. Furthermore, the complementary properties between EEG and eye movements are observed. Our code is available at https://github.com/935963004/MAET.
Highly Effective End-to-End Single-to-Multichannel Feature Fusion and Ensemble Classification to Decode Emotional Secretes from Small-Scale Spontaneous Facial Micro-Expressions

Sajjatul Islam,Yongsheng Sang,Adam A. Q. Mohammed,Lei Yuan,Jiancheng Lv,MD Sajjatul Islam

DOI: https://doi.org/10.1016/j.jksuci.2023.101653

IF: 9.006

2023-07-19

Journal of King Saud University - Computer and Information Sciences

Abstract:Facial Micro-Expression (ME) is one of the pre-dominating non-verbal clues to demystify the true emotional states that people try to conceal cautiously. But emotion recognition from spontaneous ME images limits the high accuracy due to short duration and low intensity, lack of sufficient samples and consistencies among publicly available ME datasets. In this study, we have proposed two highly effective, lightweight, and generalized single-channel DLRRF-MER, and multi-channel DLH-3C-FUSION fusion models inspired by deep dense convolutional models and texture-based feature descriptors Local Binary Pattern (LBP) and Histogram of Oriented Gradients (HOG) to recognize ME from apex frame by constructing a composite dataset from five publicly available ME datasets CASME, CASMEII, CAS(ME) 2 , SAMM and MMEW. Pre-training has been done on a new composition of five facial macro expressions datasets CK+, MUGFE, OuluCasia, SFEW, and RAF-DB. The proposed models are fine-tuned on the target ME dataset rigorously with Stratified 5-Fold and 10-Fold, Leave-One-Subject-Out, and Leave-One-Dataset-Out(LODO) cross-validations(CV). In all evaluations, our proposed algorithms show remarkable improvement in effectiveness which surpasses the state-of-the-art accuracies and results in higher generalization capacity. To the best of our knowledge, it is the first attempt to use LODO CV in ME recognition from five ME datasets.

computer science, information systems

Multi-view Emotional Expressions Dataset

Multi-view Emotional Expressions Dataset Using 2D Pose Estimation.

4DME: A Spontaneous 4D Micro-Expression Dataset with Multimodalities

M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database

MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation

MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition

HEU Emotion: A Large-scale Database for Multi-modal Emotion Recognition in the Wild

An EEG-Based Multi-Modal Emotion Database with Both Posed and Authentic Facial Actions for Emotion Analysis

Multi-View Common Space Learning For Emotion Recognition In The Wild

Mfermeta++ : Robust Multiview Facial Expression Recognition Based on Metahuman and Metalearning

Emo3D: Metric and Benchmarking Dataset for 3D Facial Expression Generation from Emotion Description

EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes

Construction and Validation of the Dalian Emotional Movement Open-Source Set (DEMOS)

HEU Emotion: a Large-Scale Database for Multimodal Emotion Recognition in the Wild

More is Better: A Database for Spontaneous Micro-Expression with High Frame Rates

MEDA-CBLSTM: Data Acquisition, Processing and Emotion Recognition Based on Multi-Layer EDA

Beihang University Facial Expression Database and Multiple Facial Expression Recognition

SEED-VII: A Multimodal Dataset of Six Basic Emotions with Continuous Labels for Emotion Recognition

MEDIC: A Multimodal Empathy Dataset in Counseling

Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels

Highly Effective End-to-End Single-to-Multichannel Feature Fusion and Ensemble Classification to Decode Emotional Secretes from Small-Scale Spontaneous Facial Micro-Expressions