Abstract:Abstract Standard registration algorithms need to be independently applied to each surface to register, following careful pre-processing and hand-tuning. Recently, learning-based approaches have emerged that reduce the registration of new scans to running inference with a previously-trained model. The potential benefits are multifold: inference is typically orders of magnitude faster than solving a new instance of a difficult optimization problem, deep learning models can be made robust to noise and corruption, and the trained model may be re-used for other tasks, e.g. through transfer learning. In this paper, we cast the registration task as a surface-to-surface translation problem, and design a model to reliably capture the latent geometric information directly from raw 3D face scans. We introduce Shape-My-Face (SMF), a powerful encoder-decoder architecture based on an improved point cloud encoder, a novel visual attention mechanism, graph convolutional decoders with skip connections, and a specialized mouth model that we smoothly integrate with the mesh convolutions. Compared to the previous state-of-the-art learning algorithms for non-rigid registration of face scans, SMF only requires the raw data to be rigidly aligned (with scaling) with a pre-defined face template. Additionally, our model provides topologically-sound meshes with minimal supervision, offers faster training time, has orders of magnitude fewer trainable parameters, is more robust to noise, and can generalize to previously unseen datasets. We extensively evaluate the quality of our registrations on diverse data. We demonstrate the robustness and generalizability of our model with in-the-wild face scans across different modalities, sensor types, and resolutions. Finally, we show that, by learning to register scans, SMF produces a hybrid linear and non-linear morphable model. Manipulation of the latent space of SMF allows for shape generation, and morphing applications such as expression transfer in-the-wild. We train SMF on a dataset of human faces comprising 9 large-scale databases on commodity hardware.

ShapeMaker: Self-Supervised Joint Shape Canonicalization, Segmentation, Retrieval and Deformation

ShapeMatcher: Self-Supervised Joint Shape Canonicalization, Segmentation, Retrieval and Deformation

Neural Shape Mating: Self-Supervised Object Assembly with Adversarial Shape Priors

Shape-Former: Bridging CNN and Transformer via ShapeConv for multimodal image matching

A Unified Feature Representation and Learning Framework for 3D Shape

Shared Latent Membership Enables Joint Shape Abstraction and Segmentation With Deformable Superquadrics

Multi-part shape matching by simultaneous partial functional correspondence

Deformation-driven Shape Correspondence Via Shape Recognition

Self-supervised Learning of Implicit Shape Representation with Dense Correspondence for Deformable Objects

Shape-Space Deformer: Unified Visuo-Tactile Representations for Robotic Manipulation of Deformable Objects

KP-RED: Exploiting Semantic Keypoints for Joint 3D Shape Retrieval and Deformation

Deformation-Guided Unsupervised Non-Rigid Shape Matching

Shape My Face: Registering 3D Face Scans by Surface-to-Surface Translation

Shape Matching And Retrieval Via Contour Multi-Scale Decomposition

Learning Manifold Patch-Based Representations of Man-Made Shapes

AShapeFormer: Semantics-Guided Object-Level Active Shape Encoding for 3D Object Detection Via Transformers

TailorMe: Self-Supervised Learning of an Anatomically Constrained Volumetric Human Shape Model

TailorMe: Self‐Supervised Learning of an Anatomically Constrained Volumetric Human Shape Model

Category-Level 6-D Object Pose Estimation With Shape Deformation for Robotic Grasp Detection

Unsupervised Shape and Pose Disentanglement for 3D Meshes

DeepShape: Deep-Learned Shape Descriptor for 3D Shape Retrieval