Encoding of multi-modal emotional information via personalized skin-integrated wireless facial interface

Jin Pyo Lee,Hanhyeok Jang,Yeonwoo Jang,Hyeonseo Song,Suwoo Lee,Pooi See Lee,Jiyun Kim

DOI: https://doi.org/10.1038/s41467-023-44673-2

IF: 16.6

2024-01-15

Nature Communications

Abstract:Abstract Human affects such as emotions, moods, feelings are increasingly being considered as key parameter to enhance the interaction of human with diverse machines and systems. However, their intrinsically abstract and ambiguous nature make it challenging to accurately extract and exploit the emotional information. Here, we develop a multi-modal human emotion recognition system which can efficiently utilize comprehensive emotional information by combining verbal and non-verbal expression data. This system is composed of personalized skin-integrated facial interface (PSiFI) system that is self-powered, facile, stretchable, transparent, featuring a first bidirectional triboelectric strain and vibration sensor enabling us to sense and combine the verbal and non-verbal expression data for the first time. It is fully integrated with a data processing circuit for wireless data transfer allowing real-time emotion recognition to be performed. With the help of machine learning, various human emotion recognition tasks are done accurately in real time even while wearing mask and demonstrated digital concierge application in VR environment.

multidisciplinary sciences

What problem does this paper attempt to address?

The paper aims to address the issue of accurate extraction and utilization of human emotional information in human-computer interaction. Specifically:

1. **Background Issue**: Human emotional parameters such as emotions and feelings are becoming increasingly important in human-computer interaction. However, their abstract and complex nature makes it very challenging to accurately extract and utilize this emotional information.

2. **Solution**: The paper proposes a multimodal human emotion recognition system (PSiFI), which efficiently utilizes comprehensive emotional information by combining verbal and non-verbal expression data. This system consists of a personalized skin-integrated facial interface (PSiFI) that is self-powered, convenient, stretchable, and transparent, and it is the first to combine verbal and non-verbal expression data. Additionally, the system integrates data processing circuits, supports wireless data transmission, and can perform emotion recognition in real-time contexts.

3. **Technical Means**: The research employs machine learning methods, especially Convolutional Neural Networks (CNN), to quickly adapt to individual contexts through transfer learning. The collected data is encoded into personalized feedback parameters, enabling real-time emotion recognition tasks. This approach not only improves the accuracy of data collection but also enhances the user experience, effectively working even when the user is wearing a mask.

In summary, the paper primarily addresses the problem of accurately capturing and recognizing complex emotional states under various environmental factors, providing new possibilities for future human-computer interaction.