Abstract:Background: Emotions affect our mental health: they influence our perception, alter our physical strength, and interfere with our reason. Emotions modulate our face, voice, and movements. When emotions are expressed through the voice or face, they are difficult to measure because cameras and microphones are not often used in real life in the same laboratory conditions where emotion detection algorithms perform well. With the increasing use of smartphones, the fact that we touch our phones, on average, thousands of times a day, and that emotions modulate our movements, we have an opportunity to explore emotional patterns in passive expressive touches and detect emotions, enabling us to empower smartphone apps with emotional intelligence. Objective: In this study, we asked 2 questions. (1) As emotions modulate our finger movements, will humans be able to recognize emotions by only looking at passive expressive touches? (2) Can we teach machines how to accurately recognize emotions from passive expressive touches? Methods: We were interested in 8 emotions: anger, awe, desire, fear, hate, grief, laughter, love (and no emotion). We conducted 2 experiments with 2 groups of participants: good imagers and emotionally aware participants formed group A, with the remainder forming group B. In the first experiment, we video recorded, for a few seconds, the expressive touches of group A, and we asked group B to guess the emotion of every expressive touch. In the second experiment, we trained group A to express every emotion on a force-sensitive smartphone. We then collected hundreds of thousands of their touches, and applied feature selection and machine learning techniques to detect emotions from the coordinates of participant' finger touches, amount of force, and skin area, all as functions of time. Results: We recruited 117 volunteers: 15 were good imagers and emotionally aware (group A); the other 102 participants formed group B. In the first experiment, group B was able to successfully recognize all emotions (and no emotion) with a high 83.8% (769/918) accuracy: 49.0% (50/102) of them were 100% (450/450) correct and 25.5% (26/102) were 77.8% (182/234) correct. In the second experiment, we achieved a high 91.11% (2110/2316) classification accuracy in detecting all emotions (and no emotion) from 9 spatiotemporal features of group A touches. Conclusions: Emotions modulate our touches on force-sensitive screens, and humans have a natural ability to recognize other people's emotions by watching prerecorded videos of their expressive touches. Machines can learn the same emotion recognition ability and do better than humans if they are allowed to continue learning on new data. It is possible to enable force-sensitive screens to recognize users' emotions and share this emotional insight with users, increasing users' emotional awareness and allowing researchers to design better technologies for well-being.

Discrete Emotions Discovered by Contactless Measurement of Facial Blood Flows

The Hybrid Discrete–dimensional Frame Method for Emotional Film Selection

Facial-Video-Based Physiological Signal Measurement: Recent Advances and Affective Applications

Non-contact Emotion Analysis Based on Videos.

Classification of emotional states via transdermal cardiovascular spatiotemporal facial patterns using multispectral face videos

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Real-Time Movie-Induced Discrete Emotion Recognition from EEG Signals

Facial Thermal and Blood Perfusion Patterns of Human Emotions: Proof-of-Concept

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning

The blenderFace method: video-based measurement of raw movement data during facial expressions of emotion using open-source software

Transdermal Optical Imaging Revealed Different Spatiotemporal Patterns of Facial Cardiovascular Activities.

The physiological measurement and evaluation of empathy of video content

Fnirs Evidence for Recognizably Different Positive Emotions

A Noncontact Emotion Recognition Method Based on Complexion and Heart Rate

Quantification of Vascular Function Changes under Different Emotion States: A Pilot Study.

Concordance Between Facial Micro-Expressions and Physiological Signals under Emotion Elicitation

Classification of Individual’s discrete emotions reflected in facial microexpressions using electroencephalogram and facial electromyogram

An Investigation of Olfactory-Enhanced Video on EEG-Based Emotion Recognition

Recognition of Emotions Conveyed by Touch Through Force-Sensitive Screens: Observational Study of Humans and Machine Learning Techniques

Sensing emotional valence and arousal dynamics through automated facial action unit analysis

Your blush gives you away: detecting hidden mental states with remote photoplethysmography and thermal imaging