Abstract:We have recently seen significant advancements in the development of robotic machines that are designed to assist people with their daily lives. Socially assistive robots are now able to perform a number of tasks autonomously and without human supervision. However, if these robots are to be accepted by human users, there is a need to focus on the form of human–robot interaction that is seen as acceptable by such users. In this paper, we extend our previous work, originally presented in Ruiz-Garcia et al. (in: Engineering applications of neural networks: 17th international conference, EANN 2016, Aberdeen, UK, September 2–5, 2016, proceedings, pp 79–93, 2016. 10.1007/978-3-319-44188-7_6), to provide emotion recognition from human facial expressions for application on a real-time robot. We expand on previous work by presenting a new hybrid deep learning emotion recognition model and preliminary results using this model on real-time emotion recognition performed by our humanoid robot. The hybrid emotion recognition model combines a Deep Convolutional Neural Network (CNN) for self-learnt feature extraction and a Support Vector Machine (SVM) for emotion classification. Compared to more complex approaches that use more layers in the convolutional model, this hybrid deep learning model produces state-of-the-art classification rate of 96.26%\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$96.26\%$$\end{document}, when tested on the Karolinska Directed Emotional Faces dataset (Lundqvist et al. in The Karolinska Directed Emotional Faces—KDEF, 1998), and offers similar performance on unseen data when tested on the Extended Cohn–Kanade dataset (Lucey et al. in: Proceedings of the third international workshop on CVPR for human communicative behaviour analysis (CVPR4HB 2010), San Francisco, USA, pp 94–101, 2010). This architecture also takes advantage of batch normalisation (Ioffe and Szegedy in Batch normalization: accelerating deep network training by reducing internal covariate shift. http://arxiv.org/abs/1502.03167, 2015) for fast learning from a smaller number of training samples. A comparison between Gabor filters and CNN for feature extraction, and between SVM and multilayer perceptron for classification is also provided.

Deep Learning Based Video Spatio-Temporal Modeling for Emotion Recognition

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Emotion Recognition in User‐generated Videos with Long‐range Correlation‐aware Network

Human emotion recognition by analyzing facial expressions, heart rate and blogs using deep learning method

Spatio-Temporal Encoder-Decoder Fully Convolutional Network for Video-Based Dimensional Emotion Recognition

Emotion Recognition from Large-Scale Video Clips with Cross-Attention and Hybrid Feature Weighting Neural Networks

Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks

A hybrid deep learning neural approach for emotion recognition from facial expressions for socially assistive robots

Human Emotion Recognition Based on Spatio-Temporal Facial Features Using HOG-HOF and VGG-LSTM

Long Short Term Memory Recurrent Neural Network Based Encoding Method for Emotion Recognition in Video.

Deep learning for affective computing: Text-based emotion recognition in decision support

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning

Multimodal emotion recognition based on a fusion of audiovisual information with temporal dynamics

Video-based driver emotion recognition using hybrid deep spatio-temporal feature learning

Spatiotemporal Networks for Video Emotion Recognition

Survey of deep emotion recognition in dynamic data using facial, speech and textual cues

Learning Expression Features via Deep Residual Attention Networks for Facial Expression Recognition From Video Sequences

Mutual Information-Based Emotion Recognition

Emotion Recognition from Time-Frequency Analysis in EEG Signals Using a Deep Learning Strategy

Efficient Spatial Temporal Convolutional Features for Audiovisual Continuous Affect Recognition