Automatic Micro-Expression Analysis: Open Challenges.

Guoying Zhao,Xiaobai Li

DOI: https://doi.org/10.3389/fpsyg.2019.01833

IF: 3.8

2019-01-01

Frontiers in Psychology

Abstract:OPINION article Front. Psychol., 07 August 2019 | https://doi.org/10.3389/fpsyg.2019.01833

What problem does this paper attempt to address?

Facial Micro-Expressions: an Overview

Guoying Zhao,Xiaobai Li,Yante Li,Matti Pietikainen

DOI: https://doi.org/10.1109/jproc.2023.3275192

IF: 20.6

2023-01-01

Proceedings of the IEEE

Abstract:Micro-expression (ME) is an involuntary, fleeting, and subtle facial expression. It may occur in high-stake situations when people attempt to conceal or suppress their true feelings. Therefore, MEs can provide essential clues to people's true feelings and have plenty of potential applications, such as national security, clinical diagnosis, and interrogations. In recent years, ME analysis has gained much attention in various fields due to its practical importance, especially automatic ME analysis in computer vision as MEs are difficult to process by naked eyes. In this survey, we provide a comprehensive review of ME development in the field of computer vision, from the ME studies in psychology and early attempts in computer vision to various computational ME analysis methods and future directions. Four main tasks in ME analysis are specifically discussed, including ME spotting, ME recognition, ME action unit detection, and ME generation in terms of the approaches, advance developments, and challenges. Through this survey, readers can understand MEs in both aspects of psychology and computer vision, and apprehend the future research direction in ME analysis.
A Review on Facial Micro-Expressions Analysis: Datasets, Features and Metrics

Walied Merghani,Adrian K. Davison,Moi Hoon Yap

DOI: https://doi.org/10.48550/arXiv.1805.02397

2018-05-07

Computer Vision and Pattern Recognition

Abstract:Facial micro-expressions are very brief, spontaneous facial expressions that appear on the face of humans when they either deliberately or unconsciously conceal an emotion. Micro-expression has shorter duration than macro-expression, which makes it more challenging for human and machine. Over the past ten years, automatic micro-expressions recognition has attracted increasing attention from researchers in psychology, computer science, security, neuroscience and other related disciplines. The aim of this paper is to provide the insights of automatic micro-expressions and recommendations for future research. There has been a lot of datasets released over the last decade that facilitated the rapid growth in this field. However, comparison across different datasets is difficult due to the inconsistency in experiment protocol, features used and evaluation methods. To address these issues, we review the datasets, features and the performance metrics deployed in the literature. Relevant challenges such as the spatial temporal settings during data collection, emotional classes versus objective classes in data labelling, face regions in data analysis, standardisation of metrics and the requirements for real-world implementation are discussed. We conclude by proposing some promising future directions to advancing micro-expressions research.
An Overview of Facial Micro-Expression Analysis: Data, Methodology and Challenge

Hong-Xia Xie,Ling Lo,Hong-Han Shuai,Wen-Huang Cheng

DOI: https://doi.org/10.48550/arXiv.2012.11307

2020-12-21

Abstract:Facial micro-expressions indicate brief and subtle facial movements that appear during emotional communication. In comparison to macro-expressions, micro-expressions are more challenging to be analyzed due to the short span of time and the fine-grained changes. In recent years, micro-expression recognition (MER) has drawn much attention because it can benefit a wide range of applications, e.g. police interrogation, clinical diagnosis, depression analysis, and business negotiation. In this survey, we offer a fresh overview to discuss new research directions and challenges these days for MER tasks. For example, we review MER approaches from three novel aspects: macro-to-micro adaptation, recognition based on key apex frames, and recognition based on facial action units. Moreover, to mitigate the problem of limited and biased ME data, synthetic data generation is surveyed for the diversity enrichment of micro-expression data. Since micro-expression spotting can boost micro-expression analysis, the state-of-the-art spotting works are also introduced in this paper. At last, we discuss the challenges in MER research and provide potential solutions as well as possible directions for further investigation.

Computer Vision and Pattern Recognition,Machine Learning
Automatic Facial Expression Analysis

Huachun Tan,Yujin Zhang

DOI: https://doi.org/10.4018/978-1-59140-562-7.ch010

2006-01-01

Abstract:Facial expression analysis is an active area in human-computer interaction. Many techniques of facial expression analysis have been proposed that try to make the interaction tighter and more efficient.
Needle in a Haystack: Spotting and recognising micro-expressions “in the wild”

Y.S. Gan,John See,Huai-Qian Khor,Kun-Hong Liu,Sze-Teng Liong

DOI: https://doi.org/10.1016/j.neucom.2022.06.101

IF: 6

2022-09-07

Neurocomputing

Abstract:Computational research on facial micro-expressions has long focused on videos captured under constrained laboratory conditions due to the challenging elicitation process and limited samples that are publicly available. Moreover, processing micro-expressions is extremely challenging under unconstrained scenarios. This paper introduces, for the first time, a completely automatic micro-expression “spot-and-recognize” framework that is performed on in-the-wild videos, such as in poker games and political interviews. The proposed method first spots the apex frame from a video by handling head movements and unconscious actions which are typically larger in motion intensity, with alignment employed to enforce a canonical face pose. Optical flow guided features play a central role in our method: they can robustly identify the location of the apex frame, and are used to learn a shallow neural network model for emotion classification. Experimental results demonstrate the feasibility of the proposed methodology, establishing good baselines for both spotting and recognition tasks – ASR of 0.33 and F1-score of 0.6758 respectively on the MEVIEW micro-expression database. In addition, we present comprehensive qualitative and quantitative analyses to further show the effectiveness of the proposed framework, with new suggestion for an appropriate evaluation protocol. In a nutshell, this paper provides a new benchmark for apex spotting and emotion recognition in an in-the-wild setting.

computer science, artificial intelligence
Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms

Xianye Ben,Yi Ren,Junping Zhang,Su-Jing Wang,Kidiyo Kpalma,Weixiao Meng,Yong-Jin Liu

DOI: https://doi.org/10.1109/tpami.2021.3067464

IF: 23.6

2021-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Unlike the conventional facial expressions, micro-expressions are involuntary and transient facial expressions capable of revealing the genuine emotions that people attempt to hide. Therefore, they can provide important information in a broad range of applications such as lie detection, criminal detection, etc. Since micro-expressions are transient and of low intensity, however, their detection and recognition is difficult and relies heavily on expert experiences. Due to its intrinsic particularity and complexity, video-based micro-expression analysis is attractive but challenging, and has recently become an active area of research. Although there have been numerous developments in this area, thus far there has been no comprehensive survey that provides researchers with a systematic overview of these developments with a unified evaluation. Accordingly, in this survey paper, we first highlight the key differences between macro- and micro-expressions, then use these differences to guide our research survey of video-based micro-expression analysis in a cascaded structure, encompassing the neuropsychological basis, datasets, features, spotting algorithms, recognition algorithms, applications and evaluation of state-of-the-art approaches. For each aspect, the basic techniques, advanced developments and major challenges are addressed and discussed. Furthermore, after considering the limitations of existing micro-expression datasets, we present and release a new dataset — called micro-and-macro expression warehouse (MMEW) — containing more video samples and more labeled emotion types. We then perform a unified comparison of representative methods on CAS(ME)<span class="mjpage"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1.054ex" height="2.343ex" style="vertical-align: -0.171ex;" viewBox="0 -934.9 453.9 1008.6" role="img" focusable="false" xmlns="http://www.w3.org/2000/svg"><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"> <use transform="scale(0.707)" xlink:href="#MJMAIN-32" x="0" y="513"></use></g></svg></span>2 for spotting, and on MMEW and SAMM for -ecognition, respectively. Finally, some potential future research directions are explored and outlined.<svg xmlns="http://www.w3.org/2000/svg" style="display: none;"><defs id="MathJax_SVG_glyphs"><path stroke-width="1" id="MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path></defs></svg>

computer science, artificial intelligence,engineering, electrical & electronic
MEGC2022

Jingting Li,Moi Hoon Yap,Wen-Huang Cheng,John See,Xiaopeng Hong,Xiaobai Li,Sujing Wang,Adrian K. Davison,Yante Li,Zizhao Dong

DOI: https://doi.org/10.1145/3503161.3551601

2022-01-01

Abstract:Facial micro-expressions (MEs) are involuntary movements of the face that occur spontaneously when a person experiences an emotion but attempts to suppress or repress the facial expression, typically found in a high-stakes environment. Unfortunately, the small sample problem severely limits the automation of ME analysis. Furthermore, due to the brief and subtle nature of ME, ME spotting is a challenging task, and the performance is still not satisfactory yet. This challenge focuses on two tasks, i.e., the micro- and macro-expression spotting task, and the ME Generation task.
Data Leakage and Evaluation Issues in Micro-Expression Analysis

Tuomas Varanka,Yante Li,Wei Peng,Guoying Zhao

DOI: https://doi.org/10.1109/taffc.2023.3265063

IF: 13.99

2023-01-01

IEEE Transactions on Affective Computing

Abstract:Micro-expressions have drawn increasing interest lately due to various potential applications. The task is, however, difficult as it incorporates many challenges from the fields of computer vision, machine learning and emotional sciences. Due to the spontaneous and subtle characteristics of micro-expressions, the available training and testing data are limited, which make evaluation complex. We show that data leakage and fragmented evaluation protocols are issues among the micro-expression literature. We find that fixing data leaks can drastically reduce model performance, in some cases even making the models perform similarly to a random classifier. To this end, we go through common pitfalls, propose a new standardized evaluation protocol using facial action units with over 2000 micro-expression samples, and provide an open source library that implements the evaluation protocols in a standardized manner. Code is publicly available in https://github.com/tvaranka/meb.

computer science, cybernetics, artificial intelligence
FME '22: 2ndWorkshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression Analysis

Jingting Li,Moi Hoon Yap,Wen-Huang Cheng,John See,Xiaopeng Hong,Xiaobai Li,Su-Jing Wang

DOI: https://doi.org/10.1145/3503161.3554777

2022-01-01

Abstract:Micro-expressions are facial movements that are extremely short and not easily detected, which often reflect the genuine emotions of individuals. Micro-expressions are important cues for understanding real human emotions and can be used for non-contact non-perceptual deception detection, or abnormal emotion recognition. It has broad application prospects in national security, judicial practice, health prevention, clinical practice, etc. However, micro-expression feature extraction and learning are highly challenging because micro-expressions have the characteristics of short duration, low intensity, and local asymmetry. In addition, the intelligent micro-expression analysis combined with deep learning technology is also plagued by the problem of small samples. Not only is micro-expression elicitation very difficult, micro-expression annotation is also very time-consuming and laborious. More importantly, the micro-expression generation mechanism is not yet clear, which shackles the application of micro-expressions in real scenarios. FME'22 is the inaugural workshop in this area of research, with the aim of promoting interactions between researchers and scholars from within this niche area of research and also including those from broader, general areas of expression and psychology research. The complete FME'22 workshop proceedings are available at: https://dl.acm.org/doi/proceedings/10.1145/3552465.
A Survey of Automatic Facial Micro-expression Analysis: Databases, Methods and Challenges

Yee-Hui Oh,John See,Anh Cat Le Ngo,Raphael Chung-Wei Phan,Vishnu Monn Baskaran

DOI: https://doi.org/10.3389/fpsyg.2018.01128

2018-06-15

Abstract:Over the last few years, automatic facial micro-expression analysis has garnered increasing attention from experts across different disciplines because of its potential applications in various fields such as clinical diagnosis, forensic investigation and security systems. Advances in computer algorithms and video acquisition technology have rendered machine analysis of facial micro-expressions possible today, in contrast to decades ago when it was primarily the domain of psychiatrists where analysis was largely manual. Indeed, although the study of facial micro-expressions is a well-established field in psychology, it is still relatively new from the computational perspective with many interesting problems. In this survey, we present a comprehensive review of state-of-the-art databases and methods for micro-expressions spotting and recognition. Individual stages involved in the automation of these tasks are also described and reviewed at length. In addition, we also deliberate on the challenges and future directions in this growing field of automatic facial micro-expression analysis.

Computer Vision and Pattern Recognition,Multimedia
FME '22: 2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression Analysis

Jingting Li,Moi Hoon Yap,Wen-Huang Cheng,John See,Xiaopeng Hong,Xiaobai Li,Su-Jing Wang

DOI: https://doi.org/10.1145/3503161.3554777

2022-01-01

Abstract:ABSTRACTMicro-expressions are facial movements that are extremely short and not easily detected, which often reflect the genuine emotions of individuals. Micro-expressions are important cues for understanding real human emotions and can be used for non-contact non-perceptual deception detection, or abnormal emotion recognition. It has broad application prospects in national security, judicial practice, health prevention, clinical practice, etc. However, micro-expression feature extraction and learning are highly challenging because micro-expressions have the characteristics of short duration, low intensity, and local asymmetry. In addition, the intelligent micro-expression analysis combined with deep learning technology is also plagued by the problem of small samples. Not only is micro-expression elicitation very difficult, micro-expression annotation is also very time-consuming and laborious. More importantly, the micro-expression generation mechanism is not yet clear, which shackles the application of micro-expressions in real scenarios. FME'22 is the inaugural workshop in this area of research, with the aim of promoting interactions between researchers and scholars from within this niche area of research and also including those from broader, general areas of expression and psychology research. The complete FME'22 workshop proceedings are available at: https://dl.acm.org/doi/proceedings/10.1145/3552465.
A Spontaneous Micro-expression Database: Inducement, Collection and Baseline

Xiaobai Li,Tomas Pfister,Xiaohua Huang,Guoying Zhao,Matti Pietikainen

DOI: https://doi.org/10.1109/fg.2013.6553717

2013-01-01

Abstract:Micro-expressions are short, involuntary facial expressions which reveal hidden emotions. Micro-expressions are important for understanding humans' deceitful behavior. Psychologists have been studying them since the 1960's. Currently the attention is elevated in both academic fields and in media. However, while general facial expression recognition (FER) has been intensively studied for years in computer vision, little research has been done in automatically analyzing micro-expressions. The biggest obstacle to date has been the lack of a suitable database. In this paper we present a novel Spontaneous Micro-expression Database SMIC, which includes 164 micro-expression video clips elicited from 16 participants. Micro-expression detection and recognition performance are provided as baselines. SMIC provides sufficient source material for comprehensive testing of automatic systems for analyzing micro-expressions, which has not been possible with any previously published database.
Facial Microexpression Recognition: A Survey

Feng XU,Jun-Ping ZHANG

DOI: https://doi.org/10.16383/j.aas.2017.c160398

2017-01-01

Abstract:Facial expression is an important channel in social interaction. Reading facial expression can improve un-derstanding of psychological condition and emotional status. Different from normal expressions, microexpression is a special kind of subtle facial action. It serves as a vital clue for affective estimation, and has broad applications in public security and psychological treatment. Recognizing microexpression requires professional training for human due to its short duration and subtle movement. So far a low recognition accuracy has been reported in the literature. In recent years, researchers have been studying microexpression recognition based on computer vision, which can largely improve the feasibility of such recognition. In this article, we introduce problem definition and current research status of microex-pression, survey several representative techniques in this topic, and discuss some underlying issues and potential research directions.
MEGC2020 - the Third Facial Micro-Expression Grand Challenge

Jingting Li,Su-Jing Wang,Moi Hoon Yap,John See,Xiaopeng Hong,Xiaobai Li

DOI: https://doi.org/10.1109/fg47880.2020.00035

2020-01-01

Abstract:The recent emergence of automatic facial micro-expression analysis has attracted a lot of attention in the last five years. Compared to the advances made in micro-expression recognition, the task of micro-expression spotting from long videos is tremendously in need of more effective methods. This paper summarises the 3rd Facial Micro-Expression Grand Challenge (MEGC 2020) held in conjunction with the 15th IEEE Conference on Automatic Face and Gesture Recognition (FG) 2020. In this workshop, we propose a new challenge of spotting both macro- and micro-expressions from long videos, to spur the community to develop new techniques for micro-expression spotting and also to extend facial micro-expression analysis to more complex real-world scenarios where micro-expressions are likely to be intertwined among normal expressions. In this paper, we outline the evaluation protocols for the challenge task, and describe the datasets involved. Then, we summarize the methods from the accepted challenge papers, present the comparison and analysis of results, as well as future directions.
Review of Micro-Expression Spotting and Recognition in Video Sequences

Hang Pan,Lun Xie,Zhiliang Wang,Bin Liu,Minghao Yang,Jianhua Tao

DOI: https://doi.org/10.1016/j.vrih.2020.10.003

2021-01-01

Virtual Reality & Intelligent Hardware

Abstract:Facial micro-expressions are short and imperceptible expressions that involuntarily reveal the true emotions that a person may be attempting to suppress, hide, disguise, or conceal. Such expressions can reflect a person's real emotions and have a wide range of application in public safety and clinical diagnosis. The analysis of facial micro-expressions in video sequences through computer vision is still relatively recent. In this research, a comprehensive review on the topic of spotting and recognition used in microexpression analysis databases and methods, is conducted, and advanced technologies in this area are summarized. In addition, we discuss challenges that remain unresolved alongside future work to be completed in the field of micro-expression analysis.
Casme Ii: an Improved Spontaneous Micro-Expression Database and the Baseline Evaluation

Wen-Jing Yan,Xiaobai Li,Su-Jing Wang,Guoying Zhao,Yong-Jin Liu,Yu-Hsin Chen,Xiaolan Fu

DOI: https://doi.org/10.1371/journal.pone.0086041

IF: 3.7

2014-01-01

PLoS ONE

Abstract:A robust automatic micro-expression recognition system would have broad applications in national safety, police interrogation, and clinical diagnosis. Developing such a system requires high quality databases with sufficient training samples which are currently not available. We reviewed the previously developed micro-expression databases and built an improved one (CASME II), with higher temporal resolution (200 fps) and spatial resolution (about 280×340 pixels on facial area). We elicited participants' facial expressions in a well-controlled laboratory environment and proper illumination (such as removing light flickering). Among nearly 3000 facial movements, 247 micro-expressions were selected for the database with action units (AUs) and emotions labeled. For baseline evaluation, LBP-TOP and SVM were employed respectively for feature extraction and classifier with the leave-one-subject-out cross-validation method. The best performance is 63.41% for 5-class classification.
Micro-expressions: A Study of Basic Reading and The Influencing Factors on Production and Recognition

Yunxuan Yang

DOI: https://doi.org/10.54097/y71ea179

2024-03-02

Abstract:In response to nonverbal communication in psychology, micro-expression research has gained widespread attention as a relatively new but growing field. As technology advances, researchers have developed computer-based tools and software to assist in analyzing micro-expressions. These tools use facial recognition and machine learning algorithms to more accurately detect and categorize micro-expressions. This article introduces the definition and basic types of micro-expressions by integrating related literature and further analyzes the four influencing factors including gender, cultural differences, occurrence background, and psychological states on the production and recognition of micro-expressions. Due to the limited research so far, we will establish the connection with micro-expressions with the help of facial expressions or emotions. Furthermore, three perspectives on micro-expression recognition are presented, including social safety, clinical therapy, and teaching efficiency, to illustrate the application of micro-expression recognition in current life. Finally, we suggest directions for future research on micro-expressions on interdisciplinary cooperation with other fields like brain science and so on.
Application and Algorithm Analysis of Micro-Expression Recognition

Zhaoqi Cai,Shaowen Yan

DOI: https://doi.org/10.1088/1742-6596/1617/1/012077

2020-08-01

Journal of Physics: Conference Series

Abstract:Abstract Micro-expressions (MEs) are the expressions of real emotion in people’s heart. It usually happens when people want to hide their true emotions. Through the study of MEs, we can correctly speculate on the inner activities of the expressors, which has important application value in the fields of criminal investigation, arrest, psychotherapy, education and teaching. At present, more and more people are paying attention to the research of ME recognition. People try to detect MEs by faster algorithms, but the recognition process is affected by many factors, so it is necessary to optimize related technologies as soon as possible to achieve higher accuracy. Therefore, this paper analyzes the application of ME recognition and related algorithms, and puts forward the development prospects.
Sliding Window Based Micro-expression Spotting: A Benchmark.

Thuong-Khanh Tran,Xiaopeng Hong,Guoying Zhao

DOI: https://doi.org/10.1007/978-3-319-70353-4_46

2017-01-01

Abstract:Micro-expressions are very rapid and involuntary facial expressions, which indicate the suppressed or concealed emotions and can lead to many potential applications. Recently, research in micro-expression spotting obtains increasing attention. By investigating existing methods, we realize that evaluation standards of micro-expression spotting methods are highly desired. To address this issue, we construct a benchmark for fairer and better performance evaluation of micro-expression spotting approaches. Firstly, we propose a sliding window based multi-scale evaluation standard with a series of protocols. Secondly, baseline results of popular features are provided. Finally, we also raise the concerns of taking advantages of machine learning techniques.
Recognising Spontaneous Facial Micro-Expressions

Tomas Pfister,Xiaobai Li,Guoying Zhao,Matti Pietikainen

DOI: https://doi.org/10.1109/iccv.2011.6126401

2011-01-01

Vision

Abstract:Facial micro-expressions are rapid involuntary facial expressions which reveal suppressed affect. To the best knowledge of the authors, there is no previous work that successfully recognises spontaneous facial micro-expressions. In this paper we show how a temporal interpolation model together with the first comprehensive spontaneous micro-expression corpus enable us to accurately recognise these very short expressions. We designed an induced emotion suppression experiment to collect the new corpus using a high-speed camera. The system is the first to recognise spontaneous facial micro-expressions and achieves very promising results that compare favourably with the human micro-expression detection accuracy.

Automatic Micro-Expression Analysis: Open Challenges.

Facial Micro-Expressions: an Overview

A Review on Facial Micro-Expressions Analysis: Datasets, Features and Metrics

An Overview of Facial Micro-Expression Analysis: Data, Methodology and Challenge

Automatic Facial Expression Analysis

Needle in a Haystack: Spotting and recognising micro-expressions “in the wild”

Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms

MEGC2022

Data Leakage and Evaluation Issues in Micro-Expression Analysis

FME '22: 2ndWorkshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression Analysis

A Survey of Automatic Facial Micro-expression Analysis: Databases, Methods and Challenges

FME '22: 2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression Analysis

A Spontaneous Micro-expression Database: Inducement, Collection and Baseline

Facial Microexpression Recognition: A Survey

MEGC2020 - the Third Facial Micro-Expression Grand Challenge

Review of Micro-Expression Spotting and Recognition in Video Sequences

Casme Ii: an Improved Spontaneous Micro-Expression Database and the Baseline Evaluation

Micro-expressions: A Study of Basic Reading and The Influencing Factors on Production and Recognition

Application and Algorithm Analysis of Micro-Expression Recognition

Sliding Window Based Micro-expression Spotting: A Benchmark.

Recognising Spontaneous Facial Micro-Expressions