Abstract:Dynamic facial expression recognition (FER) databases provide important data support for affective computing and applications. However, most FER databases are annotated with several basic mutually exclusive emotional categories and contain only one modality, e.g., videos. The monotonous labels and modality cannot accurately imitate human emotions and fulfill applications in the real world. In this paper, we propose MAFW, a large-scale multi-modal compound affective database with 10,045 video-audio clips in the wild. Each clip is annotated with a compound emotional category and a couple of sentences that describe the subjects' affective behaviors in the clip. For the compound emotion annotation, each clip is categorized into one or more of the 11 widely-used emotions, i.e., anger, disgust, fear, happiness, neutral, sadness, surprise, contempt, anxiety, helplessness, and disappointment. To ensure high quality of the labels, we filter out the unreliable annotations by an Expectation Maximization (EM) algorithm, and then obtain 11 single-label emotion categories and 32 multi-label emotion categories. To the best of our knowledge, MAFW is the first in-the-wild multi-modal database annotated with compound emotion annotations and emotion-related captions. Additionally, we also propose a novel Transformer-based expression snippet feature learning method to recognize the compound emotions leveraging the expression-change relations among different emotions and modalities. Extensive experiments on MAFW database show the advantages of the proposed method over other state-of-the-art methods for both uni- and multi-modal FER. Our MAFW database is publicly available from <a class="link-external link-https" href="https://mafw-database.github.io/MAFW" rel="external noopener nofollow">this https URL</a>.

Building a Chinese Natural Emotional Audio-Visual Database

CHEAVD: a Chinese natural emotional audio–visual database

Searching Audio-Visual Clips for Dual-mode Chinese Emotional Speech Database

Construction and Evaluation of Mandarin Multimodal Emotional Speech Database

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

The Mandarin Chinese auditory emotions stimulus database: A validated set of Chinese pseudo-sentences

HEU Emotion: A Large-scale Database for Multi-modal Emotion Recognition in the Wild

HEU Emotion: a Large-Scale Database for Multimodal Emotion Recognition in the Wild

Construction and application of chinese emotional corpus

CNAMD Corpus: A Chinese Natural Audiovisual Multimodal Database of Conversations for Social Interactive Agents

MASC: A Speech Corpus in Mandarin for Emotion Analysis and Affective Speaker Recognition

CHAD: a chinese affective database

EmoSpeech: A Corpus of Emotionally Rich and Contextually Detailed Speech Annotations

Multimodal Spontaneous Emotion Corpus for Human Behavior Analysis

MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation

An EEG-Based Multi-Modal Emotion Database with Both Posed and Authentic Facial Actions for Emotion Analysis

A standardized database of Chinese emotional short videos based on age and gender differences

MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition

Design, construction and evaluation of emotional multimodal pathological speech database

Establishment of Chinese Speech Emotion Database of Broadcasting

MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild