Multi-Modal Hierarchical Empathetic Framework for Social Robots With Affective Body Control

Yue Gao,Yangqing Fu,Ming Sun,Feng Gao

DOI: https://doi.org/10.1109/taffc.2024.3356511

IF: 13.99

2024-01-01

IEEE Transactions on Affective Computing

Abstract:Social robots require the ability to understand human emotions and provide affective and behavioral responses during human-robot interactions. However, current social robots lack empathy capabilities. In this work, we propose a novel Multi-modal Hierarchical Empathetic (MHE) framework for generating empathetic responses for social robots. MHE is composed of a multi-modal fusion and emotion recognition module, an empathetic dialogue generation module, and an expression generation module. By fusing the sensor signals of different modalities, the robot can recognize human emotions and generate affective responses. Multiple experiments are conducted on a real robot, Pepper, to evaluate the proposed framework. The experiments are conducted to discriminate between MHE-generated text and human responses in complete ignorance, and most experimenters agree that MHE can effectively generate human-like and empathetic responses. To better evaluate the similarity between human-robot and human-human interactions, a period eye movement map (PEM) captured by an eye tracker is proposed. The experimental results demonstrate the improvement in the MHE in human-robot interactions by comparing different PEMs.

computer science, cybernetics, artificial intelligence