Abstract:Affective behaviors enable social robots to not only establish better connections with humans but also serve as a tool for the robots to express their internal states. It has been well established that emotions are important to signal understanding in Human-Robot Interaction (HRI). This work aims to harness the power of Large Language Models (LLM) and proposes an approach to control the affective behavior of robots. By interpreting emotion appraisal as an Emotion Recognition in Conversation (ERC) tasks, we used GPT-3.5 to predict the emotion of a robot's turn in real-time, using the dialogue history of the ongoing conversation. The robot signaled the predicted emotion using facial expressions. The model was evaluated in a within-subjects user study (N = 47) where the model-driven emotion generation was compared against conditions where the robot did not display any emotions and where it displayed incongruent emotions. The participants interacted with the robot by playing a card sorting game that was specifically designed to evoke emotions. The results indicated that the emotions were reliably generated by the LLM and the participants were able to perceive the robot's emotions. It was found that the robot expressing congruent model-driven facial emotion expressions were perceived to be significantly more human-like, emotionally appropriate, and elicit a more positive impression. Participants also scored significantly better in the card sorting game when the robot displayed congruent facial expressions. From a technical perspective, the study shows that LLMs can be used to control the affective behavior of robots reliably in real-time. Additionally, our results could be used in devising novel human-robot interactions, making robots more effective in roles where emotional interaction is important, such as therapy, companionship, or customer service.

Visuo-auditory Multimodal Emotional Structure to Improve Human-Robot-Interaction

Dynamic Emotion Understanding in Human–Robot Interaction Based on Two-Layer Fuzzy SVR-TS Model

Multi-Modal Based Fuzzy Atmosfield in Human-Robot Interaction

A Multimodal Emotional Communication Based Humans-Robots Interaction System

A MultiModal Social Robot Toward Personalized Emotion Interaction

Disambiguating Affective Stimulus Associations for Robot Perception and Dialogue

Emotional States Based 3-D Fuzzy Atmosfield for Casual Communication Between Humans and Robots

Human-Robot Emotional Interaction Model Based on Reinforcement Learning

Multimodal Integration of Emotional Signals from Voice, Body, and Context: Effects of (In)Congruence on Emotion Recognition and Attitudes Towards Robots

An emotion-driven and topic-aware dialogue framework for human–robot interaction

Evaluation of Robot Emotion Expressions for Human–Robot Interaction

Creating Expressive Social Robots That Convey Symbolic and Spontaneous Communication

Emotion Detection for Social Robots Based on NLP Transformers and an Emotion Ontology

ExpressionBot: An Emotive Lifelike Robotic Face for Face-to-Face Communication

Emotion Recognition for Human-Robot Interaction: Recent Advances and Future Perspectives

Robust Audiovisual Emotion Recognition: Aligning Modalities, Capturing Temporal Information, and Handling Missing Features

Real-time emotion generation in human-robot dialogue using large language models

Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition

Interactive Robot Learning for Multimodal Emotion Recognition.

Data-driven emotional body language generation for social robotics

Multi-Platform Intelligent System for Multimodal Human-Computer Interaction