Automatic Facial Expression Analysis

Huachun Tan,Yujin Zhang

DOI: https://doi.org/10.4018/978-1-59140-562-7.ch010

2006-01-01

Abstract:Facial expression analysis is an active area in human-computer interaction. Many techniques of facial expression analysis have been proposed that try to make the interaction tighter and more efficient.

What problem does this paper attempt to address?

Cgan Based Facial Expression Recognition for Human-Robot Interaction

Jia Deng,Gaoyang Pang,Zhiyu Zhang,Zhibo Pang,Huayong Yang,Geng Yang

DOI: https://doi.org/10.1109/access.2019.2891668

IF: 3.9

2019-01-01

IEEE Access

Abstract:As an emerging research topic for proximity service (ProSe), automatic emotion recognition enables the machines to understand the emotional changes of human beings which can not only facilitate natural, effective, seamless, and advanced human–robot interaction or human–computer interface but also promote emotional health. Facial expression recognition (FER) is a vital task for emotion recognition. However, significant gap between human and machine exists in FER task. In this paper, we present a conditional generative adversarial network-based approach to alleviate the intra-class variations by individually controlling the facial expressions and learning the generative and discriminative representations simultaneously. The proposed framework consists of a generator G and three discriminators ( Di , Da , and Dexp ). The generator G transforms any query face image into another prototypic facial expression image with other factors preserved. Armed with action units condition, the generator G pays more attention to information relevant to facial expression. Three loss functions ( $L_{I}$ , La , and Lexp ) corresponding to the three discriminators ( Di , Da , and Dexp ) were designed to learn generative and discriminative representations. Moreover, after rendering the generated expression back to its original facial expression, cycle consistency loss is also applied to guarantee the identity and produce more constrained visual representations. Optimized by combining both synthesis and classification loss functions, the learnt representation is explicitly disentangled from other variations such as identity, head pose, and illumination. Qualitative and quantitative experimental results demonstrate the proposed FER system is effective for expression recognition.
Automatic Micro-Expression Analysis: Open Challenges.

Guoying Zhao,Xiaobai Li

DOI: https://doi.org/10.3389/fpsyg.2019.01833

IF: 3.8

2019-01-01

Frontiers in Psychology

Abstract:OPINION article Front. Psychol., 07 August 2019 | https://doi.org/10.3389/fpsyg.2019.01833
A Survey of Automatic Facial Micro-expression Analysis: Databases, Methods and Challenges

Yee-Hui Oh,John See,Anh Cat Le Ngo,Raphael Chung-Wei Phan,Vishnu Monn Baskaran

DOI: https://doi.org/10.3389/fpsyg.2018.01128

2018-06-15

Abstract:Over the last few years, automatic facial micro-expression analysis has garnered increasing attention from experts across different disciplines because of its potential applications in various fields such as clinical diagnosis, forensic investigation and security systems. Advances in computer algorithms and video acquisition technology have rendered machine analysis of facial micro-expressions possible today, in contrast to decades ago when it was primarily the domain of psychiatrists where analysis was largely manual. Indeed, although the study of facial micro-expressions is a well-established field in psychology, it is still relatively new from the computational perspective with many interesting problems. In this survey, we present a comprehensive review of state-of-the-art databases and methods for micro-expressions spotting and recognition. Individual stages involved in the automation of these tasks are also described and reviewed at length. In addition, we also deliberate on the challenges and future directions in this growing field of automatic facial micro-expression analysis.

Computer Vision and Pattern Recognition,Multimedia
Face Expression Recognition and Analysis: The State of the Art

Vinay Bettadapura

DOI: https://doi.org/10.48550/arXiv.1203.6722

2012-03-30

Abstract:The automatic recognition of facial expressions has been an active research topic since the early nineties. There have been several advances in the past few years in terms of face detection and tracking, feature extraction mechanisms and the techniques used for expression classification. This paper surveys some of the published work since 2001 till date. The paper presents a time-line view of the advances made in this field, the applications of automatic face expression recognizers, the characteristics of an ideal system, the databases that have been used and the advances made in terms of their standardization and a detailed summary of the state of the art. The paper also discusses facial parameterization using FACS Action Units (AUs) and MPEG-4 Facial Animation Parameters (FAPs) and the recent advances in face detection, tracking and feature extraction methods. Notes have also been presented on emotions, expressions and facial features, discussion on the six prototypic expressions and the recent studies on expression classifiers. The paper ends with a note on the challenges and the future work. This paper has been written in a tutorial style with the intention of helping students and researchers who are new to this field.

Computer Vision and Pattern Recognition
The Human Facial Combined Expression Recognition System

JIN Hui,GAO Wen

DOI: https://doi.org/10.3321/j.issn:0254-4164.2000.06.006

2000-01-01

Chinese Journal of Computers

Abstract:The human beings have not only rational thinking and logic reasoning ability, but also perceptual thinking and emotions. In the aim of the natural and intelligent human machine communication, it is essential to make it possible that the computer can understand and express the emotions. This paper summarizes the study in the psychology field and the former work and proposes the combined expression recognition system based on the analysis of the dynamic expression image sequences. The face is taken as being composed of several primary expression regions, in which the motion features can be extracted and constituted to eigen sequences. The analysis of the arbitrary length of image sequences of facial expressions and combined expression recognition are proposed and implemented by analyzing the respective expression meaning and the expression contents of different primary regions and using the multi feature fusion.
A Comparison of Methods of Facial Expression Recognition

Yingjie Chen,Han Wu,Tao Wang,Yizhou Wang

DOI: https://doi.org/10.1109/wrc-sara.2018.8584202

2018-01-01

Abstract:Emotional recognition based on facial expressions is a very active research topic in the field of human-computer interaction. There has been a great body of work with in-depth of study in this area. In this paper, we analyze and compare the state-of-the-art facial expression recognition methods, propose some evaluation dimensions and discuss possible directions for future research.
Facial Micro-Expressions: an Overview

Guoying Zhao,Xiaobai Li,Yante Li,Matti Pietikainen

DOI: https://doi.org/10.1109/jproc.2023.3275192

IF: 20.6

2023-01-01

Proceedings of the IEEE

Abstract:Micro-expression (ME) is an involuntary, fleeting, and subtle facial expression. It may occur in high-stake situations when people attempt to conceal or suppress their true feelings. Therefore, MEs can provide essential clues to people's true feelings and have plenty of potential applications, such as national security, clinical diagnosis, and interrogations. In recent years, ME analysis has gained much attention in various fields due to its practical importance, especially automatic ME analysis in computer vision as MEs are difficult to process by naked eyes. In this survey, we provide a comprehensive review of ME development in the field of computer vision, from the ME studies in psychology and early attempts in computer vision to various computational ME analysis methods and future directions. Four main tasks in ME analysis are specifically discussed, including ME spotting, ME recognition, ME action unit detection, and ME generation in terms of the approaches, advance developments, and challenges. Through this survey, readers can understand MEs in both aspects of psychology and computer vision, and apprehend the future research direction in ME analysis.
Automatic facial expression recognition on a single 3D face by exploring shape deformation.

Boqing Gong,Yueming Wang,Jianzhuang Liu,Xiaoou Tang

DOI: https://doi.org/10.1145/1631272.1631358

2009-01-01

Abstract:ABSTRACTFacial expression recognition has many applications in multimedia processing and the development of 3D data acquisition techniques makes it possible to identify expressions using 3D shape information. In this paper, we propose an automatic facial expression recognition approach based on a single 3D face. The shape of an expressional 3D face is approximated as the sum of two parts, a basic facial shape component (BFSC) and an expressional shape component (ESC). The BFSC represents the basic face structure and neutral-style shape and the ESC contains shape changes caused by facial expressions. To separate the BFSC and ESC, our method firstly builds a reference face for each input 3D non-neutral face by a learning method, which well represents the basic facial shape. Then, based on the BFSC and the original expressional face, a facial expression descriptor is designed. The surface depth changes are considered in the descriptor. Finally, the descriptor is input into an SVM to recognize the expression. Unlike previous methods which recognize a facial expression with the help of manually labeled key points and/or a neutral face, our method works on a single 3D face without any manual assistance. Extensive experiments are carried out on the BU-3DFE database and comparisons with existing methods are conducted. The experimental results show the effectiveness of our method.
Facial gesture interfaces for expression and communication

Michael J. Lyons

DOI: https://doi.org/10.48550/arXiv.2010.01567

IF: 6.4588

2020-10-04

Human-Computer Interaction

Abstract:Considerable effort has been devoted to the automatic extraction of information about action of the face from image sequences. Within the context of human-computer interaction (HCI) we may distinguish systems that allow expression from those which aim at recognition. Most of the work in facial action processing has been directed at automatically recognizing affect from facial actions. By contrast, facial gesture interfaces, which respond to deliberate facial actions, have received comparatively little attention. This paper reviews several projects on vision-based interfaces that rely on facial action for intentional HCI. Applications to several domains are introduced, including text entry, artistic and musical expression and assistive technology for motor-impaired users.
Facial Expression Recognition Based on Facial Action Unit

Jiannan Yang,Fan Zhang,Bike Chen,Samee U. Khan

DOI: https://doi.org/10.1109/IGSC48788.2019.8957163

2019-01-01

Abstract:In the past few years, there has been increasing interest in the perception of human expressions and mental states by machines, and Facial Expression Recognition (FER) has attracted increasing attention. Facial Action Unit (AU) is an early proposed method to describe facial muscle movements, which can effectively reflect the changes in people's facial expressions. In this paper, we propose a high-performance facial expression recognition method based on facial action unit, which can run on low-configuration computer and realize video and real-time camera FER. Our method is mainly divided into two parts. In the first part, 68 facial landmarks and image Histograms of Oriented Gradients (HOG) are obtained, and the feature values of action units are calculated accordingly. The second part uses three classification methods to realize the mapping from AUs to FER. We have conducted many experiments on the popular human FER benchmark datasets (CK+ and Oulu CASIA) to demonstrate the effectiveness of our method.
Human Emotion Identification Based on Facial Expression using Image Processing

Ayushi Jain

DOI: https://doi.org/10.17762/turcomat.v10i2.13621

2019-09-10

Turkish Journal of Computer and Mathematics Education (TURCOMAT)

Abstract:The field of using image processing for automated human emotion identification based on facial expression is one that is rapidly growing and has many potential uses. There may be many advantages to understanding human emotions, such as better human-computer interaction and more accurate diagnosis of mental illness. Understanding emotions can help you communicate more effectively with others. Traditional methods of emotion recognition rely on human interpretation, which may be expensive, time-consuming, and prone to error. Recently though, advances in computer vision and machine learning have made it possible to design fully automated systems for identifying emotions based on facial expressions. These algorithms are competent in this area because of their analysis of facial expressions. This research provides a systematic review of previous work on emotion analysis from photographs of people's faces. Issues, boundaries, and potential directions for further study are all discussed. We also explore the many suggested methods for emotion recognition and provide a thorough evaluation of the existing datasets. The applications of AI-based emotion detection are also discussed, along with the need for objective standards of evaluation and verification of the results. Our research leads us to the conclusion that automated emotion recognition algorithms have great potential as a tool for enhancing human-computer interaction, mental health assessment, and other applications, but that more work needs to be done to improve their accuracy and robustness.
A Facial Expression Emotion Recognition Based Human-robot Interaction System

Zhentao Liu,Min Wu,Weihua Cao,Luefeng Chen,Jianping Xu,Ri Zhang,Mengtian Zhou,Junwei Mao

DOI: https://doi.org/10.1109/jas.2017.7510622

2017-01-01

IEEE/CAA Journal of Automatica Sinica

Abstract:A facial expression emotion recognition based human-robot interaction（FEER-HRI） system is proposed, for which a four-layer system framework is designed. The FEERHRI system enables the robots not only to recognize human emotions, but also to generate facial expression for adapting to human emotions. A facial emotion recognition method based on2D-Gabor, uniform local binary pattern（LBP） operator, and multiclass extreme learning machine（ELM） classifier is presented,which is applied to real-time facial expression recognition for robots. Facial expressions of robots are represented by simple cartoon symbols and displayed by a LED screen equipped in the robots, which can be easily understood by human. Four scenarios,i.e., guiding, entertainment, home service and scene simulation are performed in the human-robot interaction experiment, in which smooth communication is realized by facial expression recognition of humans and facial expression generation of robots within 2 seconds. As a few prospective applications, the FEERHRI system can be applied in home service, smart home, safe driving, and so on.
Bodily Expression For Automatic Affect Recognition

Hatice Gunes,Caifeng Shan,Shizhi Chen,Yingli Tian

DOI: https://doi.org/10.1002/9781118910566.ch14

2015-01-01

Abstract:This chapter focuses on the why, what, and how of bodily expression analysis for automatic affect recognition. It first asks the question of 'why bodily expression?' and attempts to find answers by reviewing the latest bodily expression perception literature. The chapter then turns its attention to the question of 'what are the bodily expressions recognized automatically?' by providing an overview of the automatic bodily expression recognition literature. The chapter then provides representative answers to how bodily expression analysis can aid affect recognition by describing three case studies: (1) data acquisition and annotation of the first publicly available database of affective face-and-body displays (i.e., the FABO database); (2) a representative approach for affective state recognition from face-and-body display by detecting the space-time interest points in video and using Canonical Correlation Analysis (CCA) for fusion, and (3) a representative approach for explicit detection of the temporal phases (segments) of affective states (start/end of the expression and its subdivision into phases such as neutral, onset, apex, and offset) from bodily expressions. The chapter concludes by summarizing the main challenges faced and discussing how we can advance the state of the art in the field.
Analysis and Calculation on Human-Computer Interaction System of Emotion

YANG Jie,ZHAO Qiang

DOI: https://doi.org/10.3969/j.issn.1001-3695.2012.07.099

2012-01-01

Abstract:Especially facial expression and gesture image existed in a large number of interference information disadvantages,resulting in affective computing accuracy down,this paper presented a from the user's facial expressions and gestures in a systematic analysis of the emotional cues of method.First analysis of facial expressions and gestures were the feelings of the rich interactive basic part of the system,and then the nonverbal cued algorithm to judge user emotional state.At the same time,these papers extracted from an image sequence and expression of related characteristics,through the intelligent rule system analysis of the user's emotional state,and reflected the user's suit.Finally based on the main interface technology,it treated such as frustration or anger and other specific emotional states.The experimental results show that,the proposed method can accurately analyze and calculate the user's emotional information.
Video-based Facial Animation Techniques

Yueting Zhuang,Yunhe Pan,Jun Xiao

2008-01-01

Abstract:Facial expression creation is an indispensable part of human animation. As a special branch of computer graphics and image processing, facial expression synthesis has been an attractive topic for over 30 years and has appeared in many applications, e.g., complex human-computer-interface, interactive game, multimedia teaching, remote experience of virtual reality, and computer animation. Facial animation techniques have evolved from traditional key-frame technique to current techniques of image warping, video-based animation, physics-based animation and behavior animation. Related research domain includes computer graphics, artificial intelligence and machine learning. The requirement of generating high quality facial animation is more and more urgent. The traditional literalness and rigid facial animation cannot satisfy people’s need, thus it is necessary to develop high-quality and high …
Fully Automatic Upper Facial Action Recognition.

A Kapoor,Y Qi,RW Picard

DOI: https://doi.org/10.1109/amfg.2003.1240843

2003-01-01

Abstract:We provide a new fully automatic framework to analyze facial action units, the fundamental building blocks of facial expression enumerated in Paul Ekman's facial action coding system (FACS). The action units examined here include upper facial muscle movements such as inner eyebrow raise, eye widening, and so forth, which combine to form facial expressions. Although prior methods have obtained high recognition rates for recognizing facial action units, these methods either use manually preprocessed image sequences or require human specification of facial features; thus, they have exploited substantial human intervention. We present a fully automatic method, requiring no such human specification. The system first robustly detects the pupils using an infrared sensitive camera equipped with infrared LEDs. For each frame, the pupil positions are used to localize and normalize eye and eyebrow regions, which are analyzed using PCA to recover parameters that relate to the shape of the facial features. These parameters are used as input to classifiers based on support vector machines to recognize upper facial action units and all their possible combinations. On a completely natural dataset with lots of head movements, pose changes and occlusions, the new framework achieved a recognition accuracy of 69.3% for each individual AU and an accuracy of 62.5% for all possible AU combinations. This framework achieves a higher recognition accuracy on the Cohn-Kanade AU-coded facial expression database, which has been previously used to evaluate other facial action recognition system.
Analysis and Application of the Facial Expression Motions Based on Eigen-Flow

金辉,高文

2003-01-01

Journal of Software

Abstract:Analysis and recognition of the facial expressions play an important role in both the social society and the affective computing in the field of the computer science. There are three primary methods for the analysis of expression motive features: methods based on the facial geometrical structure features, on the definition of the expression space based on the eigen-face, and on the motion pattern matching. This paper extracts the feature regions of the expressions based on the facial physics-muscle model and evaluats the optical flow of the expression image sequences. The eigen-flow vectors can be calculated to constitute the eigen-sequences, and therefore, the expressions can be analyzed. The recognition system is implemented as an agent in the multi-perception machine and it is used as part of the video input for understanding the human body languages.
Facial emotion expressions in human-robot interaction: A survey

Niyati Rawal,Ruth Maria Stock-Homburg

DOI: https://doi.org/10.48550/arXiv.2103.07169

2022-02-08

Abstract:Facial expressions are an ideal means of communicating one's emotions or intentions to others. This overview will focus on human facial expression recognition as well as robotic facial expression generation. In the case of human facial expression recognition, both facial expression recognition on predefined datasets as well as in real-time will be covered. For robotic facial expression generation, hand-coded and automated methods i.e., facial expressions of a robot are generated by moving the features (eyes, mouth) of the robot by hand-coding or automatically using machine learning techniques, will also be covered. There are already plenty of studies that achieve high accuracy for emotion expression recognition on predefined datasets, but the accuracy for facial expression recognition in real-time is comparatively lower. In the case of expression generation in robots, while most of the robots are capable of making basic facial expressions, there are not many studies that enable robots to do so automatically. In this overview, state-of-the-art research in facial emotion expressions during human-robot interaction has been discussed leading to several possible directions for future research.

Robotics,Human-Computer Interaction
An Efficient Method For Facial Expression Recognition

xudong xie,kinman lam

DOI: https://doi.org/10.1117/12.631546

2005-01-01

Abstract:In this paper, an efficient method for human facial expression recognition is presented. Our method includes two major techniques: spatially maximum occurrence model (SMOM), which is used to describe the different facial expressions; and elastic shape-texture matching (ESTM), which is used to compute the similarity between two images. The combination of these two techniques, namely the SMOM-ESTM method, is used to classify the facial expressions. With our approach, the recognition rate based on the AR database is 94.5%.
Facial Expression Analysis under Partial Occlusion: A Survey

Ligang Zhang,Brijesh Verma,Dian Tjondronegoro,Vinod Chandran

DOI: https://doi.org/10.48550/arXiv.1802.08784

2018-02-24

Computer Vision and Pattern Recognition

Abstract:Automatic machine-based Facial Expression Analysis (FEA) has made substantial progress in the past few decades driven by its importance for applications in psychology, security, health, entertainment and human computer interaction. The vast majority of completed FEA studies are based on non-occluded faces collected in a controlled laboratory environment. Automatic expression recognition tolerant to partial occlusion remains less understood, particularly in real-world scenarios. In recent years, efforts investigating techniques to handle partial occlusion for FEA have seen an increase. The context is right for a comprehensive perspective of these developments and the state of the art from this perspective. This survey provides such a comprehensive review of recent advances in dataset creation, algorithm development, and investigations of the effects of occlusion critical for robust performance in FEA systems. It outlines existing challenges in overcoming partial occlusion and discusses possible opportunities in advancing the technology. To the best of our knowledge, it is the first FEA survey dedicated to occlusion and aimed at promoting better informed and benchmarked future work.

Automatic Facial Expression Analysis

Cgan Based Facial Expression Recognition for Human-Robot Interaction

Automatic Micro-Expression Analysis: Open Challenges.

A Survey of Automatic Facial Micro-expression Analysis: Databases, Methods and Challenges

Face Expression Recognition and Analysis: The State of the Art

The Human Facial Combined Expression Recognition System

A Comparison of Methods of Facial Expression Recognition

Facial Micro-Expressions: an Overview

Automatic facial expression recognition on a single 3D face by exploring shape deformation.

Facial gesture interfaces for expression and communication

Facial Expression Recognition Based on Facial Action Unit

Human Emotion Identification Based on Facial Expression using Image Processing

A Facial Expression Emotion Recognition Based Human-robot Interaction System

Bodily Expression For Automatic Affect Recognition

Analysis and Calculation on Human-Computer Interaction System of Emotion

Video-based Facial Animation Techniques

Fully Automatic Upper Facial Action Recognition.

Analysis and Application of the Facial Expression Motions Based on Eigen-Flow

Facial emotion expressions in human-robot interaction: A survey

An Efficient Method For Facial Expression Recognition

Facial Expression Analysis under Partial Occlusion: A Survey