Highly Effective End-to-End Single-to-Multichannel Feature Fusion and Ensemble Classification to Decode Emotional Secretes from Small-Scale Spontaneous Facial Micro-Expressions
Sajjatul Islam,Yongsheng Sang,Adam A. Q. Mohammed,Lei Yuan,Jiancheng Lv,MD Sajjatul Islam
DOI: https://doi.org/10.1016/j.jksuci.2023.101653
IF: 9.006
2023-07-19
Journal of King Saud University - Computer and Information Sciences
Abstract:Facial Micro-Expression (ME) is one of the pre-dominating non-verbal clues to demystify the true emotional states that people try to conceal cautiously. But emotion recognition from spontaneous ME images limits the high accuracy due to short duration and low intensity, lack of sufficient samples and consistencies among publicly available ME datasets. In this study, we have proposed two highly effective, lightweight, and generalized single-channel DLRRF-MER, and multi-channel DLH-3C-FUSION fusion models inspired by deep dense convolutional models and texture-based feature descriptors Local Binary Pattern (LBP) and Histogram of Oriented Gradients (HOG) to recognize ME from apex frame by constructing a composite dataset from five publicly available ME datasets CASME, CASMEII, CAS(ME) 2 , SAMM and MMEW. Pre-training has been done on a new composition of five facial macro expressions datasets CK+, MUGFE, OuluCasia, SFEW, and RAF-DB. The proposed models are fine-tuned on the target ME dataset rigorously with Stratified 5-Fold and 10-Fold, Leave-One-Subject-Out, and Leave-One-Dataset-Out(LODO) cross-validations(CV). In all evaluations, our proposed algorithms show remarkable improvement in effectiveness which surpasses the state-of-the-art accuracies and results in higher generalization capacity. To the best of our knowledge, it is the first attempt to use LODO CV in ME recognition from five ME datasets.
computer science, information systems