Hybrid Models for Facial Emotion Recognition in Children

Rafael Zimmer,Marcos Sobral,Helio Azevedo

2023-08-24

Abstract:This paper focuses on the use of emotion recognition techniques to assist psychologists in performing children's therapy through remotely robot operated sessions. In the field of psychology, the use of agent-mediated therapy is growing increasingly given recent advances in robotics and computer science. Specifically, the use of Embodied Conversational Agents (ECA) as an intermediary tool can help professionals connect with children who face social challenges such as Attention Deficit Hyperactivity Disorder (ADHD), Autism Spectrum Disorder (ASD) or even who are physically unavailable due to being in regions of armed conflict, natural disasters, or other circumstances. In this context, emotion recognition represents an important feedback for the psychotherapist. In this article, we initially present the result of a bibliographical research associated with emotion recognition in children. This research revealed an initial overview on algorithms and datasets widely used by the community. Then, based on the analysis carried out on the results of the bibliographical research, we used the technique of dense optical flow features to improve the ability of identifying emotions in children in uncontrolled environments. From the output of a hybrid model of Convolutional Neural Network, two intermediary features are fused before being processed by a final classifier. The proposed architecture was called HybridCNNFusion. Finally, we present the initial results achieved in the recognition of children's emotions using a dataset of Brazilian children.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the issue of assisting psychologists in identifying children's emotions during psychological therapy sessions through remote robot-operated conversations. Specifically, the authors aim to utilize emotion recognition technology to enhance psychologists' ability to identify children's emotions in uncontrolled environments, particularly when dealing with social challenges (such as Attention Deficit Hyperactivity Disorder, Autism Spectrum Disorder) or children who cannot participate in therapy in person due to being in regions affected by armed conflict, natural disasters, etc.

To achieve this goal, the authors first conducted a literature review related to children's emotion recognition to understand the currently widely used algorithms and datasets. Based on the results of the literature review, the authors proposed a new hybrid model architecture—HybridCNNFusion, which combines Convolutional Neural Networks (CNN) and dense optical flow features to improve the ability to recognize children's emotions in uncontrolled environments. The specific architecture of the model includes the fusion of two intermediate features, which are ultimately processed by a classifier. Preliminary experimental results indicate that the model has some effectiveness in recognizing the emotions of Brazilian children.

Overall, the main contribution of this study is the proposal of a new method suitable for children's emotion recognition, especially in uncontrolled environments, which helps psychologists to better conduct remote psychological therapy.