Abstract:Facial expression recognition plays an essential role in human conversation and human–computer interaction. Previous research studies have recognized facial expressions mainly based on 2D image processing requiring sensitive feature engineering and conventional machine learning approaches. The purpose of the present study was to recognize facial expressions by applying a new class of deep learning called geometric deep learning directly on 3D point cloud data. Two databases (Bosphorus and SIAT-3DFE) were used. The Bosphorus database includes sixty-five subjects with seven basic expressions (i.e., anger, disgust, fearness, happiness, sadness, surprise, and neutral). The SIAT-3DFE database has 150 subjects and 4 basic facial expressions (neutral, happiness, sadness, and surprise). First, preprocessing procedures such as face center cropping, data augmentation, and point cloud denoising were applied on 3D face scans. Then, a geometric deep learning model called PointNet++ was applied. A hyperparameter tuning process was performed to find the optimal model parameters. Finally, the developed model was evaluated using the recognition rate and confusion matrix. The facial expression recognition accuracy on the Bosphorus database was 69.01% for 7 expressions and could reach 85.85% when recognizing five specific expressions (anger, disgust, happiness, surprise, and neutral). The recognition rate was 78.70% with the SIAT-3DFE database. The present study suggested that 3D point cloud could be directly processed for facial expression recognition by using geometric deep learning approach. In perspectives, the developed model will be applied for facial palsy patients to guide and optimize the functional rehabilitation program.

3D Facial expression recognition using orthogonal tensor marginal fisher analysis on geometric maps

Automatic facial expression recognition on a single 3D face by exploring shape deformation.

An ICA-Based Other-Race Effect Elimination for Facial Expression Recognition.

Sparse Orthogonal Tucker Decomposition for 2D+3D Facial Expression Recognition

Video-Based Facial Expression Recognition Using Histogram Sequence of Local Gabor Binary Patterns from Three Orthogonal Planes

Fully automatic 3D facial expression recognition using differential mean curvature maps and histograms of oriented gradients

Fully automatic 3D facial expression recognition using a region-based approach

Deep Representation of Facial Geometric and Photometric Attributes for Automatic 3D Facial Expression Recognition

3D Facial Expression Recognition Based on Properties of Line Segments Connecting Facial Feature Points

Sparse and Low-rank Tucker Decomposition with Its Application to 2D+3D Facial Expression Recognition

Combining 2D Gabor and Local Binary Pattern for Facial Expression Recognition Using Extreme Learning Machine

3D Face Recognition using Mapped Depth Images

Feature Level Analysis for 3D Facial Expression Recognition.

Facial Expression Recognition Based on Incremental Isomap with Expression Weighted Distance

3D Face Recognition Via Conformal Representation

Deep and Shallow Covariance Feature Quantization for 3D Facial Expression Recognition

Expression Recognition from 3D Dynamic Faces Using Robust Spatio-Temporal Shape Features

Recognition of facial expressions based on salient geometric features and support vector machines

2D+3D Facial Expression Recognition via Discriminative Dynamic Range Enhancement and Multi-Scale Learning

Automatic 4D Facial Expression Recognition Using Dynamic Geometrical Image Network

Enhanced facial expression recognition using 3D point sets and geometric deep learning