Abstract:Facial expression recognition plays an essential role in human conversation and human–computer interaction. Previous research studies have recognized facial expressions mainly based on 2D image processing requiring sensitive feature engineering and conventional machine learning approaches. The purpose of the present study was to recognize facial expressions by applying a new class of deep learning called geometric deep learning directly on 3D point cloud data. Two databases (Bosphorus and SIAT-3DFE) were used. The Bosphorus database includes sixty-five subjects with seven basic expressions (i.e., anger, disgust, fearness, happiness, sadness, surprise, and neutral). The SIAT-3DFE database has 150 subjects and 4 basic facial expressions (neutral, happiness, sadness, and surprise). First, preprocessing procedures such as face center cropping, data augmentation, and point cloud denoising were applied on 3D face scans. Then, a geometric deep learning model called PointNet++ was applied. A hyperparameter tuning process was performed to find the optimal model parameters. Finally, the developed model was evaluated using the recognition rate and confusion matrix. The facial expression recognition accuracy on the Bosphorus database was 69.01% for 7 expressions and could reach 85.85% when recognizing five specific expressions (anger, disgust, happiness, surprise, and neutral). The recognition rate was 78.70% with the SIAT-3DFE database. The present study suggested that 3D point cloud could be directly processed for facial expression recognition by using geometric deep learning approach. In perspectives, the developed model will be applied for facial palsy patients to guide and optimize the functional rehabilitation program.

3D Facial expression recognition based on basic geometric features

A new facial feature based on the fusion of texture and shape characteristics

Automatic facial expression recognition on a single 3D face by exploring shape deformation.

3D Facial Expression Recognition Based on Properties of Line Segments Connecting Facial Feature Points

Using Feature Points and Angles Between Them to Recognise Facial Expression by a Neural Network Approach

Enhanced facial expression recognition using 3D point sets and geometric deep learning

Feature Level Analysis for 3D Facial Expression Recognition.

Deep Representation of Facial Geometric and Photometric Attributes for Automatic 3D Facial Expression Recognition

Automatic 3D Facial Expression Recognition Based on a Bayesian Belief Net and a Statistical Facial Feature Model

An Efficient Multimodal 2D + 3D Feature-Based Approach to Automatic Facial Expression Recognition

Facial Expression Recognition with LBP and ORB Features

Face Feature Extraction and Recognition Via Local Binary Pattern and Two-Dimensional Locality Preserving Projection

Point CNN:3D Face Recognition with Local Feature Descriptor and Feature Enhancement Mechanism

Facial Expression Recognition Using Hybrid Features of Pixel and Geometry

Fully automatic 3D facial expression recognition using a region-based approach

Location-sensitive sparse representation of deep normal patterns for expression-robust 3D face recognition

Fusing Normal Vector and Curvature Features on the Mesh for 3D Facial Expression Recognition.

A novel approach to expression recognition from non-frontal face images.

3D Face Recognition Based on Intrinsic Features.

A Novel Facial Feature Point Localization Method on 3D Faces

Recognition of facial expressions based on salient geometric features and support vector machines