Conveying Emotions to Robots through Touch and Sound

Qiaoqiao Ren,Remko Proesmans,Frederick Bossuyt,Jan Vanfleteren,Francis Wyffels,Tony Belpaeme

2024-12-04

Abstract:Human emotions can be conveyed through nuanced touch gestures. However, there is a lack of understanding of how consistently emotions can be conveyed to robots through touch. This study explores the consistency of touch-based emotional expression toward a robot by integrating tactile and auditory sensory reading of affective haptic expressions. We developed a piezoresistive pressure sensor and used a microphone to mimic touch and sound channels, respectively. In a study with 28 participants, each conveyed 10 emotions to a robot using spontaneous touch gestures. Our findings reveal a statistically significant consistency in emotion expression among participants. However, some emotions obtained low intraclass correlation values. Additionally, certain emotions with similar levels of arousal or valence did not exhibit significant differences in the way they were conveyed. We subsequently constructed a multi-modal integrating touch and audio features to decode the 10 emotions. A support vector machine (SVM) model demonstrated the highest accuracy, achieving 40% for 10 classes, with "Attention" being the most accurately conveyed emotion at a balanced accuracy of 87.65%.

Robotics,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to consistently convey human emotions to robots through touch and sound. Specifically, the research aims to explore the following three main issues:

1. **Are people consistent when expressing emotions through touch?**
Researchers collected data from 28 participants through experiments. Each participant used spontaneous tactile gestures to convey 10 different emotions to the robot. The study found that participants had statistically significant consistency when expressing emotions, but the intra - class correlation coefficient (ICC) of some emotions was low, indicating that there were large individual differences in the expression of these emotions.

2. **Which emotions can be clearly distinguished through touch?**
By analyzing the tactile features between different emotions, the study found that there were significant differences between most pairs of emotions, but some pairs of emotions (such as "calm" and "sad", "confused" and "comfortable", etc.) had no significant differences, indicating that these emotions were similar in tactile features and difficult to distinguish.

3. **Can a multimodal method be used to decode different emotions?**
Researchers constructed a multimodal model that combined tactile and audio features to decode 10 emotions. The results showed that the Support Vector Machine (SVM) model had the highest classification accuracy among the 10 categories, reaching 40%, and the classification accuracy of the emotion "attention" was the highest, reaching 87.65%.

Overall, this paper experimentally proves the possibility of conveying emotions to robots through touch and sound, and explores the performance and distinguishability of different emotions in tactile and sound features. These research results provide an important reference for future human - computer interaction design.