Abstract:Background: Emotions affect our mental health: they influence our perception, alter our physical strength, and interfere with our reason. Emotions modulate our face, voice, and movements. When emotions are expressed through the voice or face, they are difficult to measure because cameras and microphones are not often used in real life in the same laboratory conditions where emotion detection algorithms perform well. With the increasing use of smartphones, the fact that we touch our phones, on average, thousands of times a day, and that emotions modulate our movements, we have an opportunity to explore emotional patterns in passive expressive touches and detect emotions, enabling us to empower smartphone apps with emotional intelligence. Objective: In this study, we asked 2 questions. (1) As emotions modulate our finger movements, will humans be able to recognize emotions by only looking at passive expressive touches? (2) Can we teach machines how to accurately recognize emotions from passive expressive touches? Methods: We were interested in 8 emotions: anger, awe, desire, fear, hate, grief, laughter, love (and no emotion). We conducted 2 experiments with 2 groups of participants: good imagers and emotionally aware participants formed group A, with the remainder forming group B. In the first experiment, we video recorded, for a few seconds, the expressive touches of group A, and we asked group B to guess the emotion of every expressive touch. In the second experiment, we trained group A to express every emotion on a force-sensitive smartphone. We then collected hundreds of thousands of their touches, and applied feature selection and machine learning techniques to detect emotions from the coordinates of participant' finger touches, amount of force, and skin area, all as functions of time. Results: We recruited 117 volunteers: 15 were good imagers and emotionally aware (group A); the other 102 participants formed group B. In the first experiment, group B was able to successfully recognize all emotions (and no emotion) with a high 83.8% (769/918) accuracy: 49.0% (50/102) of them were 100% (450/450) correct and 25.5% (26/102) were 77.8% (182/234) correct. In the second experiment, we achieved a high 91.11% (2110/2316) classification accuracy in detecting all emotions (and no emotion) from 9 spatiotemporal features of group A touches. Conclusions: Emotions modulate our touches on force-sensitive screens, and humans have a natural ability to recognize other people's emotions by watching prerecorded videos of their expressive touches. Machines can learn the same emotion recognition ability and do better than humans if they are allowed to continue learning on new data. It is possible to enable force-sensitive screens to recognize users' emotions and share this emotional insight with users, increasing users' emotional awareness and allowing researchers to design better technologies for well-being.

Haptic Empathy: Conveying Emotional Meaning through Vibrotactile Feedback

Affective Haptics and Multimodal Experiments Research

Affective Haptics Research and Interaction Design.

Conveying Emotions to Robots through Touch and Sound

Hearing through Vibrations: Perception of Musical Emotions by Profoundly Deaf People

The Making of Meaning through Dyadic Haptic Affective Touch

Haptic feedback in a virtual crowd scenario improves the emotional response

An emotional expression system with vibrotactile feedback during the robot's speech

Interpersonal Transmission of Vibrotactile Feedback Via Smart Bracelets: Mechanics and Perception

Encoding of tactile information in hand via skin-integrated wireless haptic interface

Tactile emoticons: Conveying social emotions and intentions with manual and robotic tactile feedback during social media communications

The Impact of Affective Haptic Design of Wearable Devices on Persian Users

Emotional VR handshake by controlling skin deformation distribution

A Skin-Integrated Multimodal Haptic Interface for Immersive Tactile Feedback

An Evaluation Method for Hand-held Tactile Emotional Design

Subtle Contact Nuances in the Delivery of Human-to-Human Touch Distinguish Emotional Sentiment

Recognition of Emotions Conveyed by Touch Through Force-Sensitive Screens: Observational Study of Humans and Machine Learning Techniques

User Testing on Conveying Haptic Directional Information

Haptic interface with flexible self-sensing actuators for wireless touch communication

The Role of Gesture Types and Spatial Feedback in Haptic Communication

Vibrotactile enhancement of musical engagement