Abstract:Facial expression recognition plays an essential role in human conversation and human–computer interaction. Previous research studies have recognized facial expressions mainly based on 2D image processing requiring sensitive feature engineering and conventional machine learning approaches. The purpose of the present study was to recognize facial expressions by applying a new class of deep learning called geometric deep learning directly on 3D point cloud data. Two databases (Bosphorus and SIAT-3DFE) were used. The Bosphorus database includes sixty-five subjects with seven basic expressions (i.e., anger, disgust, fearness, happiness, sadness, surprise, and neutral). The SIAT-3DFE database has 150 subjects and 4 basic facial expressions (neutral, happiness, sadness, and surprise). First, preprocessing procedures such as face center cropping, data augmentation, and point cloud denoising were applied on 3D face scans. Then, a geometric deep learning model called PointNet++ was applied. A hyperparameter tuning process was performed to find the optimal model parameters. Finally, the developed model was evaluated using the recognition rate and confusion matrix. The facial expression recognition accuracy on the Bosphorus database was 69.01% for 7 expressions and could reach 85.85% when recognizing five specific expressions (anger, disgust, happiness, surprise, and neutral). The recognition rate was 78.70% with the SIAT-3DFE database. The present study suggested that 3D point cloud could be directly processed for facial expression recognition by using geometric deep learning approach. In perspectives, the developed model will be applied for facial palsy patients to guide and optimize the functional rehabilitation program.

Expression recognition from 3D dynamic faces using robust spatio-temporal shape features

Robust Face Recognition by Fusion Local Singular Value Feature and Deformable Model

Automatic facial expression recognition on a single 3D face by exploring shape deformation.

Robust 3D Face Recognition by Local Shape Difference Boosting

3D Facial Expression Recognition Based on Properties of Line Segments Connecting Facial Feature Points

Landmarks-assisted Collaborative Deep Framework for Automatic 4D Facial Expression Recognition.

3D Dynamic Facial Expression Recognition Using Low-Resolution Videos

Fully automatic 3D facial expression recognition using a region-based approach

Towards Reading Beyond Faces for Sparsity-aware 3D/4D Affect Recognition

An Efficient Multimodal 2D + 3D Feature-Based Approach to Automatic Facial Expression Recognition

Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework

Feature Level Analysis for 3D Facial Expression Recognition.

Location-sensitive sparse representation of deep normal patterns for expression-robust 3D face recognition

3D/4D Facial Expression Analysis: an Advanced Annotated Face Model Approach

Facial Expression Recognition Based on the Fusion of Spatio-temporal Features in Video Sequences

Fully automatic 3D facial expression recognition using differential mean curvature maps and histograms of oriented gradients

Deep Representation of Facial Geometric and Photometric Attributes for Automatic 3D Facial Expression Recognition

3d Face Recognition Under Expression Variations Using Similarity Metrics Fusion

Enhanced facial expression recognition using 3D point sets and geometric deep learning

3D Facial Expression Reconstruction using Cascaded Regression

A novel approach to expression recognition from non-frontal face images.