Abstract:In this paper, we present a multimodal dataset for affective computing research acquired in a human-computer interaction (HCI) setting. An experimental mobile and interactive scenario was designed and implemented based on a gamified generic paradigm for the induction of dialog-based HCI relevant emotional and cognitive load states. It consists of six experimental sequences, inducing Interest, Overload, Normal, Easy, Underload, and Frustration. Each sequence is followed by subjective feedbacks to validate the induction, a respiration baseline to level off the physiological reactions, and a summary of results. Further, prior to the experiment, three questionnaires related to emotion regulation (ERQ), emotional control (TEIQue-SF), and personality traits (TIPI) were collected from each subject to evaluate the stability of the induction paradigm. Based on this HCI scenario, the University of Ulm Multimodal Affective Corpus (uulmMAC), consisting of two homogenous samples of 60 participants and 100 recording sessions was generated. We recorded 16 sensor modalities including 4 × video, 3 × audio, and 7 × biophysiological, depth, and pose streams. Further, additional labels and annotations were also collected. After recording, all data were post-processed and checked for technical and signal quality, resulting in the final uulmMAC dataset of 57 subjects and 95 recording sessions. The evaluation of the reported subjective feedbacks shows significant differences between the sequences, well consistent with the induced states, and the analysis of the questionnaires shows stable results. In summary, our uulmMAC database is a valuable contribution for the field of affective computing and multimodal data analysis: Acquired in a mobile interactive scenario close to real HCI, it consists of a large number of subjects and allows transtemporal investigations. Validated via subjective feedbacks and checked for quality issues, it can be used for affective computing and machine learning applications.

An XML-Based Implementation of Multimodal Affective Annotation.

Modeling And Application Of Multimodal Affective User Interface With Multimedia

A color bar based affective annotation method for media player

EmoPlayer: A Media Player for Video Clips with Affective Annotations.

Recent Trends of Multimodal Affective Computing: A Survey from NLP Perspective

MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild

ASMMC-MMAC 2018: The Joint Workshop of 4th the Workshop on Affective Social Multimedia Computing and first Multi-Modal Affective Computing of Large-Scale Multimedia Data Workshop.

Multimodal Information Processing For Affective Computing

Happy Companion : A System of Multimodal Human-Computer Affective Interaction

Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment

A systematic review on affective computing: emotion models, databases, and recent advances

Affective Behaviour Analysis via Integrating Multi-Modal Knowledge

The uulmMAC Database—A Multimodal Affective Corpus for Affective Computing in Human-Computer Interaction

Multimodal Affective Computing Based on Weighted Linear Fusion

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

Multi-modal Approach for Affective Computing

End-to-end Semantic-centric Video-based Multimodal Affective Computing

A Multimodal Sentiment Analysis Approach Based on a Joint Chained Interactive Attention Mechanism

ASMMC21: the 6th International Workshop on Affective Social Multimedia Computing

Identifying affective levels on music video via completing the missing modality