Abstract:We have recently seen significant advancements in the development of robotic machines that are designed to assist people with their daily lives. Socially assistive robots are now able to perform a number of tasks autonomously and without human supervision. However, if these robots are to be accepted by human users, there is a need to focus on the form of human–robot interaction that is seen as acceptable by such users. In this paper, we extend our previous work, originally presented in Ruiz-Garcia et al. (in: Engineering applications of neural networks: 17th international conference, EANN 2016, Aberdeen, UK, September 2–5, 2016, proceedings, pp 79–93, 2016. 10.1007/978-3-319-44188-7_6), to provide emotion recognition from human facial expressions for application on a real-time robot. We expand on previous work by presenting a new hybrid deep learning emotion recognition model and preliminary results using this model on real-time emotion recognition performed by our humanoid robot. The hybrid emotion recognition model combines a Deep Convolutional Neural Network (CNN) for self-learnt feature extraction and a Support Vector Machine (SVM) for emotion classification. Compared to more complex approaches that use more layers in the convolutional model, this hybrid deep learning model produces state-of-the-art classification rate of 96.26%\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$96.26\%$$\end{document}, when tested on the Karolinska Directed Emotional Faces dataset (Lundqvist et al. in The Karolinska Directed Emotional Faces—KDEF, 1998), and offers similar performance on unseen data when tested on the Extended Cohn–Kanade dataset (Lucey et al. in: Proceedings of the third international workshop on CVPR for human communicative behaviour analysis (CVPR4HB 2010), San Francisco, USA, pp 94–101, 2010). This architecture also takes advantage of batch normalisation (Ioffe and Szegedy in Batch normalization: accelerating deep network training by reducing internal covariate shift. http://arxiv.org/abs/1502.03167, 2015) for fast learning from a smaller number of training samples. A comparison between Gabor filters and CNN for feature extraction, and between SVM and multilayer perceptron for classification is also provided.

Optimized Anfis Model with Hybrid Metaheuristic Algorithms for Facial Emotion Recognition

Adaptive neuro-fuzzy based hybrid classification model for emotion recognition from EEG signals

Automated Facial Emotion Recognition Using the Pelican Optimization Algorithm with a Deep Convolutional Neural Network

Impact of self adaptive-elephant herding optimization towards neural network for facial emotion recognition

Deep neural network-based emotion recognition using facial landmark features and particle swarm optimization

Modelling an efficient hybridized approach for facial emotion recognition using unconstraint videos and deep learning approaches

Automated facial expression recognition using exemplar hybrid deep feature generation technique

Adaptive Feature Selection-Based AdaBoost-KNN With Direct Optimization for Dynamic Emotion Recognition in Human–Robot Interaction

Hybrid CNN-SVM Classifier for Human Emotion Recognition Using ROI Extraction and Feature Fusion

Bioinspired Image Processing Enabled Facial Emotion Recognition Using Equilibrium Optimizer With a Hybrid Deep Learning Model

An efficient facial emotion recognition system using novel deep learning neural network-regression activation classifier

Deep Residual Adaptive Neural Network Based Feature Extraction for Cognitive Computing with Multimodal Sentiment Sensing and Emotion Recognition Process

Multimodal Facial Emotion Recognition Using Improved Convolution Neural Networks Model

A hybrid deep learning neural approach for emotion recognition from facial expressions for socially assistive robots

Research on Emotion Recognition with Physiological Signals Based on Hybrid Intelligent Optimization Algorithm

Optimized Ensemble Machine Learning Approach for Emotion Detection from Thermal Images

Multimodal Human Facial Emotion Recognition Using Densenet-161 and Image Feature Stabilization Algorithm

Emotion recognition framework using multiple modalities for an effective human–computer interaction

Emotion recognition through facial expression analysis based on a neurofuzzy network

Bi-Model Engagement Emotion Recognition Based on Facial and Upper-Body Landmarks and Machine Learning Approaches

Human facial emotion recognition using improved black hole based extreme learning machine