Real Time Facial Expression Classification

Wang Yubo,Ai Haizhou,Wu Bo,HUANG Chang

DOI: https://doi.org/10.3321/j.issn:1003-9775.2005.06.028

2005-01-01

Abstract:In this paper, the problem of facial expression classification is discussed,

What problem does this paper attempt to address?

WiFace: Facial Expression Recognition Using Wi-Fi Signals

Yanjiao Chen,Runmin Ou,Zhiyang Li,Kaishun Wu

DOI: https://doi.org/10.1109/tmc.2020.3001989

IF: 6.075

2020-01-01

IEEE Transactions on Mobile Computing

Abstract:Facial expressions are an essential form of human nonverbal communication. Recognition of this nonverbal sign may enable developers to understand the feedbacks on smart device functionality and advertising. Existing approaches for facial expression recognition are mainly based on cameras or on-body sensors, which are either sensitive to lighting conditions or cumbersome for users to wear devices on their faces. In this paper, we propose a new facial expression recognition system based on Wi-Fi signals, named WiFace. Our fundamental intuition is that facial muscle movements in different expressions will induce distinctive waveform patterns in the time-series of channel state information (CSI) in Wi-Fi signals. We develop a series of algorithms to process the CSI signals and extract the most representative waveform patterns for facial expression classification. We build a fully-functional prototype of WiFace using commercial off-the-shelf devices, which can recognize six typical facial expressions. We conduct extensive experiments to evaluate the performance of WiFace, and the experimental results show that the average recognition accuracy is 94.80 percent.
Real-Time facial expression recognition system based on HMM and feature point localization

Yumei Fan,Ning Cheng,Zhiliang Wang,Jiwei Liu,Changsheng Zhu

DOI: https://doi.org/10.1007/11573548_27

2005-01-01

Abstract:It is difficult for computer to recognize facial expression in real-time. Until now, no good method is put forward to solve this problem in and abroad. In this paper, we present an effective automated system that we developed to recognize facial gestures in real-time. According to psychologists’ facial expression classification, we define four basic facial expressions and localize key facial feature points exactly then extract facial components’ contours. We analyze and record facial components’ movements using information in sequential frames to recognize facial gestures. Since different facial expressions can have some same movements, it is necessary to use a good facial expression model to describe the relation between expression states and observed states of facial components’ movements for achieving a good recognition results. HMM is such a good method which can meet our requirement. We present a facial expression model based on HMM and get good real-time recognition results.
Real Time Facial Expression Recognition with AdaBoost

YB Wang,HZ Ai,B Wu,C Huang

DOI: https://doi.org/10.1109/icpr.2004.1334680

2004-01-01

Abstract:In this paper, we propose a novel method for facial expression recognition. The facial expression is extracted from human faces by an expression classifier that is learned from boosting Haar feature based look-up-table type weak classifiers. The expression recognition system consists of three modules, face detection, facial feature landmark extraction and facial expression recognition. The implemented system can automatically recognize seven expressions in real time that include anger, disgust, fear, happiness, neutral, sadness and surprise. Experimental results are reported to show its potential applications in human computer interaction.
A Comparison of Methods of Facial Expression Recognition

Yingjie Chen,Han Wu,Tao Wang,Yizhou Wang

DOI: https://doi.org/10.1109/wrc-sara.2018.8584202

2018-01-01

Abstract:Emotional recognition based on facial expressions is a very active research topic in the field of human-computer interaction. There has been a great body of work with in-depth of study in this area. In this paper, we analyze and compare the state-of-the-art facial expression recognition methods, propose some evaluation dimensions and discuss possible directions for future research.
Facial expression recognition based on deep learning

Huilin Ge,Zhiyu Zhu,Yuewei Dai,Biao Wang,Xuedong Wu

DOI: https://doi.org/10.1016/j.cmpb.2022.106621

Abstract:Background and objective: Facial expression recognition technology will play an increasingly important role in our daily life. Autonomous driving, virtual reality and all kinds of robots integrated into our life depend on the development of facial expression recognition technology. Many tasks in the field of computer vision are based on deep learning technology and convolutional neural network. The paper proposes an occluded expression recognition model based on the generated countermeasure network. The model is divided into two modules, namely, occluded face image restoration and face recognition. Methods: Firstly, this paper summarizes the research status of deep facial expression recognition methods in recent ten years and the development of related facial expression database. Then, the current facial expression recognition methods based on deep learning are divided into two categories: Static facial expression recognition and dynamic facial expression recognition. The two methodswill be introduced and summarized respectively. Aiming at the advanced deep expression recognition algorithms in the field, the performance of these algorithms on common expression databases is compared, and the strengths and weaknesses of these algorithms are analyzed in detail. Discussion and results: As the task of facial expression recognition is gradually transferred from the controlled laboratory environment to the challenging real-world environment, with the rapid development of deep learning technology, deep neural network can learn discriminative features, and is gradually applied to automatic facial expression recognition task. The current deep facial expression recognition system is committed to solve the following two problems: (1) Overfitting due to lack of sufficient training data; (2) In the real world environment, other variables that have nothing to do with expression bring interference problems. Conclusion: From the perspective of algorithm, combining other expression models, such as facial action unit model and pleasure arousal dimension model, as well as other multimodal models, such as audio mode, 3D face depth information and human physiological information, can make expression recognition more practical.
Real-Time Facial Expression Recognition Based on the Improved Brain Emotional Learning Model

Ying Mei,Xin Yang,Zhentao Liu,He Wu,Junyu Guo

DOI: https://doi.org/10.23919/chicc.2018.8483375

2018-01-01

Abstract:In this paper, a brain-inspired method is proposed for real-time facial expression recognition. The brain emotional learning (BEL) model mimics the high speed of the emotional learning mechanism in brain, it has the superior feature of fast learning and it can avoid slow convergence in traditional networks. To improve the performance of BEL model, particle swarm optimization (PSO) is applied to optimize the weights of BEL model. The integrated classifier named as PSO-BEL is tested on the well-known Cohn-Kanade database, JAFFE database, the average recognition rates reach 97.71% and 96.77%, respectively. Moreover, a real-world interactive interface is established. The comparisons of experiments with the traditional methods indicate the superiority of the proposed PSO-BEL in terms of recognition accuracy and computing time. It demonstrates that the proposed PSO-BEL can successfully meet the requirement of real-time facial expressions recognition.
A Fast and Accurate Multi-Model Facial Expression Recognition Method for Affective Intelligent Robots

Yingjie Chen,Tao Wang,Han Wu,Yizhou Wang

DOI: https://doi.org/10.1109/IISR.2018.8535644

2018-01-01

Abstract:With the development of affective computing, emotion recognition based on facial expression recognition has become an attractive topic in human-robot interaction. We hope that affective intelligent robots can detect and identify human facial expressions through vision, and make different responses accordingly. In this paper, we compare different facial expression recognition methods that use traditional machine learning models or deep learning models, and propose a fast and accurate multi-model facial expression recognition method for affective intelligence robots to complete the real-time and high-precision facial expression recognition task.
The Human Facial Combined Expression Recognition System

JIN Hui,GAO Wen

DOI: https://doi.org/10.3321/j.issn:0254-4164.2000.06.006

2000-01-01

Chinese Journal of Computers

Abstract:The human beings have not only rational thinking and logic reasoning ability, but also perceptual thinking and emotions. In the aim of the natural and intelligent human machine communication, it is essential to make it possible that the computer can understand and express the emotions. This paper summarizes the study in the psychology field and the former work and proposes the combined expression recognition system based on the analysis of the dynamic expression image sequences. The face is taken as being composed of several primary expression regions, in which the motion features can be extracted and constituted to eigen sequences. The analysis of the arbitrary length of image sequences of facial expressions and combined expression recognition are proposed and implemented by analyzing the respective expression meaning and the expression contents of different primary regions and using the multi feature fusion.
Facial expression classification using salient pattern driven integrated geometric and textual features

Ruiqi Li,Jing Tian,Matthew Chin Heng Chua

DOI: https://doi.org/10.1007/s11042-018-6133-z

IF: 2.577

2018-05-23

Multimedia Tools and Applications

Abstract:Facial expression classification aims to recognize human emotion via face images. The major challenge of facial expression classification is how to extract discriminative features from the human face images to differentiate various emotions. To tackle this challenge, a new feature extraction approach is proposed in this paper. The proposed approach defines a new set of salient patterns at the facial keypoint locations. This is in contrast to conventional approaches that either represent the whole face image as a regular grid or use local image patches centered at all facial key point locations. Driven by the proposed salient patterns, both the geometric and textual features are extracted and then concatenated and further incorporated into a machine learning framework to perform facial expression classification. The proposed approach is evaluated in the well-known CK + benchmark dataset to demonstrate its superior performance.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
A real time face emotion classification and recognition using deep learning model

Ahlam Salim Abdallah Al Balushi,Shaik Asif Hussain

DOI: https://doi.org/10.1088/1742-6596/1432/1/012087

2020-01-01

Journal of Physics: Conference Series

Abstract:Abstract Facial Detection and recognition research has been widely studied in recent years. The facial recognition applications plays an important role in many areas such as security, camera surveillance, identity verification in modern electronic devices, criminal investigations, database management systems and smart card applications etc. This work presents deep learning algorithms used in facial recognition for accurate identification and detection. The main objective of facial recognition is to authenticate and identify the facial features. However, the facial features are captured in real time and processed using haar cascade detection. The sequential process of the work is defined in three different phases where in the first phase human face is detected from the camera and in the second phase, the captured input is analyzed based on the features and database used with support of keras convolutional neural network model. In the last phase human face is authenticated to classify the emotions of human as happy, neutral, angry, sad, disgust and surprise. The proposed work presented is simplified in three objectives as face detection, recognition and emotion classification. In support of this work Open CV library, dataset and python programming is used for computer vision techniques involved. In order to prove real time efficacy, an experiment was conducted for multiple students to identify their inner emotions and find physiological changes for each face. The results of the experiments demonstrates the perfections in face analysis system. Finally, the performance of automatic face detection and recognition is measured with Accuracy.
Automatic Facial Expression Analysis

Huachun Tan,Yujin Zhang

DOI: https://doi.org/10.4018/978-1-59140-562-7.ch010

2006-01-01

Abstract:Facial expression analysis is an active area in human-computer interaction. Many techniques of facial expression analysis have been proposed that try to make the interaction tighter and more efficient.
Expression Analysis Based on Face Regions in Real-world Conditions.

Zheng Lian,Ya Li,Jian-Hua Tao,Jian Huang,Ming-Yue Niu

DOI: https://doi.org/10.1007/s11633-019-1176-9

2019-01-01

International Journal of Automation and Computing

Abstract:Facial emotion recognition is an essential and important aspect of the field of human-machine interaction. Past research on facial emotion recognition focuses on the laboratory environment. However, it faces many challenges in real-world conditions, i.e., illumination changes, large pose variations and partial or full occlusions. Those challenges lead to different face areas with different degrees of sharpness and completeness. Inspired by this fact, we focus on the authenticity of predictions generated by different pairs. For example, if only the mouth areas are available and the emotion classifier predicts happiness, then there is a question of how to judge the authenticity of predictions. This problem can be converted into the contribution of different face areas to different emotions. In this paper, we divide the whole face into six areas: nose areas, mouth areas, eyes areas, nose to mouth areas, nose to eyes areas and mouth to eyes areas. To obtain more convincing results, our experiments are conducted on three different databases: facial expression recognition + ( FER+), real-world affective faces database (RAF-DB) and expression in-the-wild (ExpW) dataset. Through analysis of the classification accuracy, the confusion matrix and the class activation map (CAM), we can establish convincing results. To sum up, the contributions of this paper lie in two areas: 1) We visualize concerned areas of human faces in emotion recognition; 2) We analyze the contribution of different face areas to different emotions in real-world conditions through experimental analysis. Our findings can be combined with findings in psychology to promote the understanding of emotional expressions.
A Facial Expression Emotion Recognition Based Human-robot Interaction System

Zhentao Liu,Min Wu,Weihua Cao,Luefeng Chen,Jianping Xu,Ri Zhang,Mengtian Zhou,Junwei Mao

DOI: https://doi.org/10.1109/jas.2017.7510622

2017-01-01

IEEE/CAA Journal of Automatica Sinica

Abstract:A facial expression emotion recognition based human-robot interaction（FEER-HRI） system is proposed, for which a four-layer system framework is designed. The FEERHRI system enables the robots not only to recognize human emotions, but also to generate facial expression for adapting to human emotions. A facial emotion recognition method based on2D-Gabor, uniform local binary pattern（LBP） operator, and multiclass extreme learning machine（ELM） classifier is presented,which is applied to real-time facial expression recognition for robots. Facial expressions of robots are represented by simple cartoon symbols and displayed by a LED screen equipped in the robots, which can be easily understood by human. Four scenarios,i.e., guiding, entertainment, home service and scene simulation are performed in the human-robot interaction experiment, in which smooth communication is realized by facial expression recognition of humans and facial expression generation of robots within 2 seconds. As a few prospective applications, the FEERHRI system can be applied in home service, smart home, safe driving, and so on.
Weight-Adapted Convolution Neural Network for Facial Expression Recognition

Min Wu,Wanjuan Su,Luefeng Chen,Zhentao Liu,Weihua Cao,Kaoru Hirota

DOI: https://doi.org/10.1109/tsmc.2019.2897330

2021-01-01

Abstract:The weight-adapted convolution neural network (WACNN) is proposed to extract discriminative expression representations for recognizing facial expression. It aims to make good use of the convolution neural network's (CNN's) potential performance in avoiding local optima and speeding up convergence by the hybrid genetic algorithm (HGA) with optimal initial population, in such a way that it realizes deep and global emotion understanding in human-robot interaction. Moreover, the idea of novelty search is introduced to solve the deception problem in the HGA, which can expend the search space to help genetic algorithm jump out of local optimum and optimize large-scale parameters. In the proposal, the facial expression image preprocessing is conducted first, then the low-level expression features are extracted by using a principal component analysis. Finally, the high-level expression semantic features are extracted and recognized by WACNN which is optimized by HGA. In order to evaluate the effectiveness of WACNN, experiments on JAFFE, CK+, and static facial expressions in the wild 2.0 databases are carried out by using k -fold cross validation, and experimental results show the recognition accuracies of the proposal are superior to that of the state-of-the-art, such as local directional ternary pattern and weighted mixture deep neural network (DNN), which aim to extract discriminative and are the DNN-based methods. Moreover, recognition accuracies of the proposal are also higher than the deep CNN without HGA, which indicates that the proposal has better global optimization ability. Meanwhile, preliminary application experiments are also carried out by using the proposed algorithm on the emotional social robot system, where nine volunteers and two-wheeled robots experience the scenario of emotion understanding. Application results indicate that the wheeled robots can recognize basic expressions, such as happy, surprise, and so on.
Evaluation of Online Teaching Quality Based on Facial Expression Recognition

Changbo Hou,Jiajun Ai,Yun Lin,Chenyang Guan,Jiawen Li,Wenyu Zhu

DOI: https://doi.org/10.3390/fi14060177

2022-06-09

Future Internet

Abstract:In 21st-century society, with the rapid development of information technology, the scientific and technological strength of all walks of life is increasing, and the field of education has also begun to introduce high and new technologies gradually. Affected by the epidemic, online teaching has been implemented all over the country, forming an education model of "dual integration" of online and offline teaching. However, the disadvantages of online teaching are also very obvious; that is, teachers cannot understand the students' listening status in real-time. Therefore, our study adopts automatic face detection and expression recognition based on a deep learning framework and other related technologies to solve this problem, and it designs an analysis system of students' class concentration based on expression recognition. The students' class concentration analysis system can help teachers detect students' class concentration and improve the efficiency of class evaluation. In this system, OpenCV is used to call the camera to collect the students' listening status in real-time, and the MTCNN algorithm is used to detect the face of the video to frame the location of the student's face image. Finally, the obtained face image is used for real-time expression recognition by using the VGG16 network added with ECANet, and the students' emotions in class are obtained. The experimental results show that the method in our study can more accurately identify students' emotions in class and carry out a teaching effect evaluation, which has certain application value in intelligent education fields, such as the smart classroom and distance learning. For example, a teaching evaluation module can be added to the teaching software, and teachers can know the listening emotions of each student in class while lecturing.

English Else
Real-time Emotion and Gender Classification using Ensemble CNN

Abhinav Lahariya,Varsha Singh,Uma Shanker Tiwary

DOI: https://doi.org/10.48550/arXiv.2111.07746

2021-11-15

Abstract:Analysing expressions on the person's face plays a very vital role in identifying emotions and behavior of a person. Recognizing these expressions automatically results in a crucial component of natural human-machine interfaces. Therefore research in this field has a wide range of applications in bio-metric authentication, surveillance systems , emotion to emoticons in various social media platforms. Another application includes conducting customer satisfaction surveys. As we know that the large corporations made huge investments to get feedback and do surveys but fail to get equitable responses. Emotion & Gender recognition through facial gestures is a technology that aims to improve product and services performance by monitoring customer behavior to specific products or service staff by their evaluation. In the past few years there have been a wide variety of advances performed in terms of feature extraction mechanisms , detection of face and also expression classification techniques. This paper is the implementation of an Ensemble CNN for building a real-time system that can detect emotion and gender of the person. The experimental results shows accuracy of 68% for Emotion classification into 7 classes (angry, fear , sad , happy , surprise , neutral , disgust) on FER-2013 dataset and 95% for Gender classification (Male or Female) on IMDB dataset. Our work can predict emotion and gender on single face images as well as multiple face images. Also when input is given through webcam our complete pipeline of this real-time system can take less than 0.5 seconds to generate results.

Computer Vision and Pattern Recognition
Face Expression Recognition and Analysis: The State of the Art

Vinay Bettadapura

DOI: https://doi.org/10.48550/arXiv.1203.6722

2012-03-30

Abstract:The automatic recognition of facial expressions has been an active research topic since the early nineties. There have been several advances in the past few years in terms of face detection and tracking, feature extraction mechanisms and the techniques used for expression classification. This paper surveys some of the published work since 2001 till date. The paper presents a time-line view of the advances made in this field, the applications of automatic face expression recognizers, the characteristics of an ideal system, the databases that have been used and the advances made in terms of their standardization and a detailed summary of the state of the art. The paper also discusses facial parameterization using FACS Action Units (AUs) and MPEG-4 Facial Animation Parameters (FAPs) and the recent advances in face detection, tracking and feature extraction methods. Notes have also been presented on emotions, expressions and facial features, discussion on the six prototypic expressions and the recent studies on expression classifiers. The paper ends with a note on the challenges and the future work. This paper has been written in a tutorial style with the intention of helping students and researchers who are new to this field.

Computer Vision and Pattern Recognition
Studies Advanced in Development and Application of Facial Expression Recognition

Piao Guo

DOI: https://doi.org/10.54097/zfcrdw98

2024-03-13

Abstract:Facial expression is an important way to convey people's inner emotional changes, as the basis of emotional understanding and the prerequisite for computers to understand human emotions, expression recognition has attracted more and more research attention from academia and industry in recent years. Most of the early expression recognition methods relied on manual features such as texture, geometry, and contour. Thanks to the powerful feature representation capability of convolutional neural networks, deep learning-based face recognition technology has made breakthroughs in recognition accuracy and speed. This paper details the latest research progress in the field of face expression recognition, including the design ideas, key steps, advantages and disadvantages of representative methods. This paper also introduces the common face expression recognition datasets and quantitatively compares the results of different methods on these datasets. Finally, this paper discusses the problems in the face expression recognition research field and look forward to the future development.
A Face Sequence Recognition Method Based on Deep Convolutional Neural Network

Siwei Ma,Meng Cao,Jiadong Li,Quanyin Zhu,Xiang Li,Yi Shen,Mengdi Wang

DOI: https://doi.org/10.1109/dcabes48411.2019.00033

2019-01-01

Abstract:Although the research in the field of face recognition has achieved high-quality results in the controllable production environment, its further research is limited in the field of face facial expression recognition based on video streaming. This paper describes the recent experimental research progress of a project, an analysis system of students' attendance status designed for real-time face recognition and prediction in class. In this paper, an expression recognition model based on mini-Xcetion optimization is proposed and the experimental results show that the facial expression recognition accuracy of this model is up to 98.96%, which is better than other algorithms in accuracy and more suitable for the application requirements of the system under the target scene studied.
Real-time Speech-Driven Animation of Expressive Talking Faces.

Jia Liu,Mingyu You,Chun Chen,Mingli Song

DOI: https://doi.org/10.1080/03081079.2010.544896

2011-01-01

International Journal of General Systems

Abstract:In this paper, we present a real-time facial animation system in which speech drives mouth movements and facial expressions synchronously. Considering five basic emotions, a hierarchical structure with an upper layer of emotion classification is established. Based on the recognized emotion label, the under-layer classification at sub-phonemic level has been modelled on the relationship between acoustic features of frames and audio labels in phonemes. Using certain constraint, the predicted emotion labels of speech are adjusted to gain the facial expression labels which are combined with sub-phonemic labels. The combinations are mapped into facial action units (FAUs), and audio-visual synchronized animation with mouth movements and facial expressions is generated by morphing between FAUs. The experimental results demonstrate that the two-layer structure succeeds in both emotion and sub-phonemic classifications, and the synthesized facial sequences reach a comparative convincing quality.

Real Time Facial Expression Classification

WiFace: Facial Expression Recognition Using Wi-Fi Signals

Real-Time facial expression recognition system based on HMM and feature point localization

Real Time Facial Expression Recognition with AdaBoost

A Comparison of Methods of Facial Expression Recognition

Facial expression recognition based on deep learning

Real-Time Facial Expression Recognition Based on the Improved Brain Emotional Learning Model

A Fast and Accurate Multi-Model Facial Expression Recognition Method for Affective Intelligent Robots

The Human Facial Combined Expression Recognition System

Facial expression classification using salient pattern driven integrated geometric and textual features

A real time face emotion classification and recognition using deep learning model

Automatic Facial Expression Analysis

Expression Analysis Based on Face Regions in Real-world Conditions.

A Facial Expression Emotion Recognition Based Human-robot Interaction System

Weight-Adapted Convolution Neural Network for Facial Expression Recognition

Evaluation of Online Teaching Quality Based on Facial Expression Recognition

Real-time Emotion and Gender Classification using Ensemble CNN

Face Expression Recognition and Analysis: The State of the Art

Studies Advanced in Development and Application of Facial Expression Recognition

A Face Sequence Recognition Method Based on Deep Convolutional Neural Network

Real-time Speech-Driven Animation of Expressive Talking Faces.