Abstract:Existing methods for 3D tracking from monocular RGB videos predominantly consider articulated and rigid objects. Modelling dense non-rigid object deformations in this setting remained largely unaddressed so far, although such effects can improve the realism of the downstream applications such as AR/VR and avatar communications. This is due to the severe ill-posedness of the monocular view setting and the associated challenges. While it is possible to naively track multiple non-rigid objects independently using 3D templates or parametric 3D models, such an approach would suffer from multiple artefacts in the resulting 3D estimates such as depth ambiguity, unnatural intra-object collisions and missing or implausible deformations. Hence, this paper introduces the first method that addresses the fundamental challenges depicted above and that allows tracking human hands interacting with human faces in 3D from single monocular RGB videos. We model hands as articulated objects inducing non-rigid face deformations during an active interaction. Our method relies on a new hand-face motion and interaction capture dataset with realistic face deformations acquired with a markerless multi-view camera system. As a pivotal step in its creation, we process the reconstructed raw 3D shapes with position-based dynamics and an approach for non-uniform stiffness estimation of the head tissues, which results in plausible annotations of the surface deformations, hand-face contact regions and head-hand positions. At the core of our neural approach are a variational auto-encoder supplying the hand-face depth prior and modules that guide the 3D tracking by estimating the contacts and the deformations. Our final 3D hand and face reconstructions are realistic and more plausible compared to several baselines applicable in our setting, both quantitatively and qualitatively. <a class="link-external link-https" href="https://vcai.mpi-inf.mpg.de/projects/Decaf" rel="external noopener nofollow">this https URL</a>

DIREG3D: DIrectly REGress 3D Hands from Multiple Cameras

In-Hand 3D Object Reconstruction from a Monocular RGB Video

Dual Regression for Efficient Hand Pose Estimation

3D head tracking using SIFT-based registration and scale invariant view-based appearance model

HandDGP: Camera-Space Hand Mesh Prediction with Differentiable Global Positioning

A Survey on 3D Hand Pose Estimation: Cameras, Methods, and Datasets

RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video

3D hand tracking for human computer interaction

Back to RGB: 3D tracking of hands and hand-object interactions based on short-baseline stereo

HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos

Decaf: Monocular Deformation Capture for Face and Hand Interactions

Dynamics Based 3D Skeletal Hand Tracking

3D Articulated Hand Tracking Based on Behavioral Model

Marker-Less 3d Human Motion Capture With Monocular Image Sequence And Height-Maps

Real-time pose and shape reconstruction of two interacting hands with a single depth camera

Joint 3D Tracking of a Deformable Object in Interaction with a Hand

Hand's 3D movement detection with one handheld camera.

3D Hand Mesh Recovery from Monocular RGB in Camera Space

Real-time Fingertip Tracking and Gesture Recognition Using RGB-D Camera

Denoising Diffusion for 3D Hand Pose Estimation from Images

Reconstructing Hand-Held Objects from Monocular Video.