Abstract:Background: Emotions affect our mental health: they influence our perception, alter our physical strength, and interfere with our reason. Emotions modulate our face, voice, and movements. When emotions are expressed through the voice or face, they are difficult to measure because cameras and microphones are not often used in real life in the same laboratory conditions where emotion detection algorithms perform well. With the increasing use of smartphones, the fact that we touch our phones, on average, thousands of times a day, and that emotions modulate our movements, we have an opportunity to explore emotional patterns in passive expressive touches and detect emotions, enabling us to empower smartphone apps with emotional intelligence. Objective: In this study, we asked 2 questions. (1) As emotions modulate our finger movements, will humans be able to recognize emotions by only looking at passive expressive touches? (2) Can we teach machines how to accurately recognize emotions from passive expressive touches? Methods: We were interested in 8 emotions: anger, awe, desire, fear, hate, grief, laughter, love (and no emotion). We conducted 2 experiments with 2 groups of participants: good imagers and emotionally aware participants formed group A, with the remainder forming group B. In the first experiment, we video recorded, for a few seconds, the expressive touches of group A, and we asked group B to guess the emotion of every expressive touch. In the second experiment, we trained group A to express every emotion on a force-sensitive smartphone. We then collected hundreds of thousands of their touches, and applied feature selection and machine learning techniques to detect emotions from the coordinates of participant' finger touches, amount of force, and skin area, all as functions of time. Results: We recruited 117 volunteers: 15 were good imagers and emotionally aware (group A); the other 102 participants formed group B. In the first experiment, group B was able to successfully recognize all emotions (and no emotion) with a high 83.8% (769/918) accuracy: 49.0% (50/102) of them were 100% (450/450) correct and 25.5% (26/102) were 77.8% (182/234) correct. In the second experiment, we achieved a high 91.11% (2110/2316) classification accuracy in detecting all emotions (and no emotion) from 9 spatiotemporal features of group A touches. Conclusions: Emotions modulate our touches on force-sensitive screens, and humans have a natural ability to recognize other people's emotions by watching prerecorded videos of their expressive touches. Machines can learn the same emotion recognition ability and do better than humans if they are allowed to continue learning on new data. It is possible to enable force-sensitive screens to recognize users' emotions and share this emotional insight with users, increasing users' emotional awareness and allowing researchers to design better technologies for well-being.

A survey on mobile affective computing

A Comprehensive Survey on Affective Computing; Challenges, Trends, Applications, and Future Directions

Affect Sensing on Smartphone - Possibilities of Understanding Cognitive Decline in Aging Population

A systematic review on affective computing: emotion models, databases, and recent advances

A Survey on Personalized Affective Computing in Human-Machine Interaction

Ubiquitous Affective Computing: A Review

A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges

Affective Game Computing: A Survey

A Review of AI Cloud and Edge Sensors, Methods, and Applications for the Recognition of Emotional, Affective and Physiological States

The Affective Dynamics of Everyday Digital Life: Opening Computational Possibility

Ieee Access Special Section Editorial: Emotion-Aware Mobile Computing

Affective Computing for eHealth Using Low-Cost Remote Internet of Things-Based EMG Platform

Recognition of Emotions Conveyed by Touch Through Force-Sensitive Screens: Observational Study of Humans and Machine Learning Techniques

Human-computer Affective Interaction System Research and Application Based on Physiological Signals: A Survey

Affective Computing for Healthcare: Recent Trends, Applications, Challenges, and Beyond

Physiological and Affective Computing through Thermal Imaging: A Survey

Analysis of infoware and software for human affective states recognition

A Review, Current Challenges, and Future Possibilities on Emotion Recognition Using Machine Learning and Physiological Signals

Recent Trends of Multimodal Affective Computing: A Survey from NLP Perspective

Affect is Computable——A Survey on Affective Computing

MoodExplorer