Abstract:We have recently seen significant advancements in the development of robotic machines that are designed to assist people with their daily lives. Socially assistive robots are now able to perform a number of tasks autonomously and without human supervision. However, if these robots are to be accepted by human users, there is a need to focus on the form of human–robot interaction that is seen as acceptable by such users. In this paper, we extend our previous work, originally presented in Ruiz-Garcia et al. (in: Engineering applications of neural networks: 17th international conference, EANN 2016, Aberdeen, UK, September 2–5, 2016, proceedings, pp 79–93, 2016. 10.1007/978-3-319-44188-7_6), to provide emotion recognition from human facial expressions for application on a real-time robot. We expand on previous work by presenting a new hybrid deep learning emotion recognition model and preliminary results using this model on real-time emotion recognition performed by our humanoid robot. The hybrid emotion recognition model combines a Deep Convolutional Neural Network (CNN) for self-learnt feature extraction and a Support Vector Machine (SVM) for emotion classification. Compared to more complex approaches that use more layers in the convolutional model, this hybrid deep learning model produces state-of-the-art classification rate of 96.26%\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$96.26\%$$\end{document}, when tested on the Karolinska Directed Emotional Faces dataset (Lundqvist et al. in The Karolinska Directed Emotional Faces—KDEF, 1998), and offers similar performance on unseen data when tested on the Extended Cohn–Kanade dataset (Lucey et al. in: Proceedings of the third international workshop on CVPR for human communicative behaviour analysis (CVPR4HB 2010), San Francisco, USA, pp 94–101, 2010). This architecture also takes advantage of batch normalisation (Ioffe and Szegedy in Batch normalization: accelerating deep network training by reducing internal covariate shift. http://arxiv.org/abs/1502.03167, 2015) for fast learning from a smaller number of training samples. A comparison between Gabor filters and CNN for feature extraction, and between SVM and multilayer perceptron for classification is also provided.

Emotion Recognition from Galvanic Skin Response Signal Based on Deep Hybrid Neural Networks

Emotion extraction based on multi bio-signal using back-propagation neural network

EEG Emotion Recognition Based on Self-attention Dynamic Graph Neural Networks

A Multi-Head Pseudo Nodes Based Spatial–temporal Graph Convolutional Network for Emotion Perception from GAIT

From face detection to emotion recognition on the framework of Raspberry pi and galvanic skin response sensor for visual and physiological biosignals

An Analysis of Emotion Recognition Based on GSR Signal

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Emotion Recognition Based on Skin Potential Signals with a Portable Wireless Device

A hybrid deep learning neural approach for emotion recognition from facial expressions for socially assistive robots

Human Emotion Recognition Based On Galvanic Skin Response signal Feature Selection and SVM

Emotion Recognition Using Electrodermal Activity Signals and Multiscale Deep Convolution Neural Network

Enhancing Emotion Detection with Non-invasive Multi-Channel EEG and Hybrid Deep Learning Architecture

Emotion recognition based on multimodal physiological signals using spiking feed-forward neural networks

Electroencephalogram Access for Emotion Recognition Based on a Deep Hybrid Network

Optimized recurrent neural network based brain emotion recognition technique

Human emotion recognition by analyzing facial expressions, heart rate and blogs using deep learning method

Deep Convolutional Neural Network For Emotion Recognition Using Eeg And Peripheral Physiological Signal

Emotion Recognition for Challenged People Facial Appearance in Social using Neural Network

A VGG16 Based Hybrid Deep Convolutional Neural Network Based Real-Time Video Frame Emotion Detection System for Affective Human Computer Interaction

Multimodal emotion recognition based on the fusion of vision, EEG, ECG, and EMG signals

Multimodal Facial Emotion Recognition Using Improved Convolution Neural Networks Model