Abstract:Human computer interaction is increasingly utilized in smart home, industry 4.0 and personal health. Communication between human and computer can benefit by a flawless exchange of emotions. As emotions have substantial influence on cognitive processes of the human brain such as learning, memory, perception and problem solving, emotional interactions benefit different applications. It can further be relevant in modern health care especially in interaction with patients suffering from stress or depression. Additionally rehabilitation applications, guiding patients through their rehabilitation training while adapting to the patients emotional state, would be highly motivating and might lead to a faster recovery. Depending on the application area, different systems for emotion recognition suit different purposes. The aim of this work is to give an overview of methods to recognize emotions and to compare their applicability based on existing studies. This review paper should enable practitioners, researchers and engineers to find a system most suitable for certain applications. An entirely contact-less method is to analyze facial features with the help of a video camera. This is useful when computers, smart-phones or tablets with integrated cameras are included in the task. Smart wearables provide contact with the skin and physiological parameters such as electro-dermal activity and heart related signals can be recorded unobtrusively also during dynamical tasks. Next to unimodal solutions, multimodal affective computing systems are analyzed since they promise higher classification accuracy. Accuracy varies based on the amount of detected emotions, extracted features, classification method and the quality of the database. Electroencephalography achieves 88.86 % accuracy for four emotions, multimodal measurements (Electrocardiography, Electromyography and bio-signals) 79.3 % for four emotive states, facial recognition 89 % for seven states and speech recognition 80.46 % for happiness and sadness. Looking forward, heart-related parameters might be an option to measure emotions accurately and unobtrusive with the help of smart wearables. This can be used in dynamic or outdoor tasks. Facial recognition on the other hand is a useful contact-less tool when it comes to emotion recognition during computer interaction.

Critical Analysis on Multimodal Emotion Recognition in Meeting the Requirements for Next Generation Human Computer Interactions

Multimodal Characterization of Emotion within Multimedia Space

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Emotion recognition framework using multiple modalities for an effective human–computer interaction

Research And Application Analysis of Multimodal Emotion Recognition Methods Based on Speech, Text, And Facial Expressions

Multimodal Emotion Recognition by Combining Physiological Signals and Facial Expressions: a Preliminary Study.

Multi-modal Approach for Affective Computing

A Review of Key Technologies for Emotion Analysis Using Multimodal Information

Multimodal emotion recognition from audio and video

Multimodal Emotion Recognition Using Different Fusion Techniques

Multimodal Affect Recognition using Kinect

Survey on AI-Based Multimodal Methods for Emotion Detection

Multimodal Emotion Recognition with Deep Learning: Advancements, challenges, and future directions

A Multimodal Emotion Sensing Platform for Building Emotion-Aware Applications

Emotion Recognition from Multiple Modalities: Fundamentals and Methodologies

Multimodal Sentiment Sensing and Emotion Recognition Based on Cognitive Computing Using Hidden Markov Model with Extreme Learning Machine

Multimodal Emotion Recognition System Using Machine Learning Classifier

Multimodal Affect Analysis for Product Feedback Assessment

Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals

Attention-based multimodal sentiment analysis and emotion recognition using deep neural networks

Emotion Recognition from Physiological Signal Analysis: A Review