Abstract:We have recently seen significant advancements in the development of robotic machines that are designed to assist people with their daily lives. Socially assistive robots are now able to perform a number of tasks autonomously and without human supervision. However, if these robots are to be accepted by human users, there is a need to focus on the form of human–robot interaction that is seen as acceptable by such users. In this paper, we extend our previous work, originally presented in Ruiz-Garcia et al. (in: Engineering applications of neural networks: 17th international conference, EANN 2016, Aberdeen, UK, September 2–5, 2016, proceedings, pp 79–93, 2016. 10.1007/978-3-319-44188-7_6), to provide emotion recognition from human facial expressions for application on a real-time robot. We expand on previous work by presenting a new hybrid deep learning emotion recognition model and preliminary results using this model on real-time emotion recognition performed by our humanoid robot. The hybrid emotion recognition model combines a Deep Convolutional Neural Network (CNN) for self-learnt feature extraction and a Support Vector Machine (SVM) for emotion classification. Compared to more complex approaches that use more layers in the convolutional model, this hybrid deep learning model produces state-of-the-art classification rate of 96.26%\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$96.26\%$$\end{document}, when tested on the Karolinska Directed Emotional Faces dataset (Lundqvist et al. in The Karolinska Directed Emotional Faces—KDEF, 1998), and offers similar performance on unseen data when tested on the Extended Cohn–Kanade dataset (Lucey et al. in: Proceedings of the third international workshop on CVPR for human communicative behaviour analysis (CVPR4HB 2010), San Francisco, USA, pp 94–101, 2010). This architecture also takes advantage of batch normalisation (Ioffe and Szegedy in Batch normalization: accelerating deep network training by reducing internal covariate shift. http://arxiv.org/abs/1502.03167, 2015) for fast learning from a smaller number of training samples. A comparison between Gabor filters and CNN for feature extraction, and between SVM and multilayer perceptron for classification is also provided.

Emotion recognition models for companion robots

A hybrid deep learning neural approach for emotion recognition from facial expressions for socially assistive robots

Emotion Recognition System via Facial Expressions and Speech Using Machine Learning and Deep Learning Techniques

Emotion Recognition for Human-Robot Interaction: Recent Advances and Future Perspectives

Real-time emotion generation in human-robot dialogue using large language models

Intelligent Facial Emotion Recognition and Semantic-Based Topic Detection for A Humanoid Robot

Optimized, robust, real-time emotion prediction for human-robot interactions using deep learning

Improving Human-Robot Interaction by Enhancing NAO Robot Awareness of Human Facial Expression

Multimodal emotion recognition from audio and video

Preliminary Study on Indonesian Word Recognition for Elder Companion Robot

Real-time emotional health detection using fine-tuned transfer networks with multimodal fusion

Study on emotion recognition and companion Chatbot using deep neural network

Human emotion recognition by analyzing facial expressions, heart rate and blogs using deep learning method

A Fast and Accurate Multi-Model Facial Expression Recognition Method for Affective Intelligent Robots

Human Face Emotion in 3D Using Machine Learning

Speech-Based Emotion Recognition using Neural Networks and Information Visualization

Deploying Machine Learning Techniques for Human Emotion Detection

AI-Assisted Emotion Recognition: Impacts on Mental Health Education and Learning Motivation

Hybrid Models for Facial Emotion Recognition in Children

Emotion recognition using machine learning: Opportunities and challenges for supporting those with autism or depression

Emotion Detection and Regulation from Personal Assistant Robot in Smart Environment