Abstract:Abstract Facial mesh tracking enables the production of topologically consistent 3D facial meshes from stereo video input captured by calibrated cameras. This technology is an integral part of many digital human applications, such as personalized avatar creation, audio-driven 3D facial animation, and talking face video generation. Currently, most facial mesh tracking methods are built on computer graphics techniques, which involve complex procedures and often necessitate human annotation within pipelines. As a result, these approaches are difficult to implement and hard to generalize across various scenarios. We propose a backpropagation-based solution that formulates facial mesh tracking as a differentiable optimization problem called the BPMT. Our solution leverages visual clues extracted from the stereo input to estimate vertex-wise geometry and texture information. The BPMT is composed of two steps: automatic face analysis and mesh tracking. In the first step, a range of visual clues are automatically extracted from the input, including facial point clouds, multi-view 2D landmarks, 3D landmarks in the world coordinate system, motion fields, and image masks. The second step can be viewed as a differentiable optimization problem, with constraints comprising stereo video input and facial clues. The primary objective is to achieve topologically consistent 3D facial meshes across frames. Additionally, the parameters to be optimized encompass the positions of free-form deformed vertices and a shared texture UV map. Furthermore, the 3D morphable model (3DMM) is introduced as a form of regularization to enhance the convergence of the optimization process. Leveraging the fully developed backpropagation software, we progressively register the facial meshes to the recorded object, generating high-quality 3D faces with consistent topologies. The BPMT requires no manual labeling within the pipeline, making it suitable for producing large-scale stereo facial data. Moreover, our method exhibits a high degree of flexibility and extensibility, positioning it as a promising platform for future research in the community.

Full Head Performance Capture Using Multi-scale Mesh Propagation

3D head tracking using SIFT-based registration and scale invariant view-based appearance model

Three-dimensional head pose tracking using registration and multi-scale appearance model

Real-time high-fidelity facial performance capture

High-precision Human Body Acquisition Via Multi-View Binocular Stereopsis

LiveCap: Real-time Human Performance Capture from Monocular Video

Look Ma, no markers: holistic performance capture without the hassle

Dynamic Human Body Reconstruction and Motion Tracking with Low-Cost Depth Cameras

Topo4D: Topology-Preserving Gaussian Splatting for High-Fidelity 4D Head Capture

Instant Multi-View Head Capture through Learnable Registration

Multi-view Human Motion Capture with an Improved Deformation Skin Model

Performance Capture of High-Speed Motion Using Staggered Multi-View Recording

Light-Weight Multi-view Topology Consistent Facial Geometry and Reflectance Capture.

Markerless motion capture of multiple characters using multiview image segmentation

Markerless Shape and Motion Capture From Multiview Video Sequences

Topologically Consistent Multi-View Face Inference Using Volumetric Sampling

HybridFusion: Real-Time Performance Capture Using a Single Depth Sensor and Sparse IMUs

MulayCap: Multi-layer Human Performance Capture Using A Monocular Video Camera

DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Cameras

Human Performance Capture Using Multiple Handheld Kinects

Formulating facial mesh tracking as a differentiable optimization problem: a backpropagation-based solution