Abstract:Reconstructing hand-held objects from monocular RGB images is an appealing yet challenging task. In this task, contacts between hands and objects provide important cues for recovering the 3D geometry of the hand-held objects. Though recent works have employed implicit functions to achieve impressive progress, they ignore formulating contacts in their frameworks, which results in producing less realistic object meshes. In this work, we explore how to model contacts in an explicit way to benefit the implicit reconstruction of hand-held objects. Our method consists of two components: explicit contact prediction and implicit shape reconstruction. In the first part, we propose a new subtask of directly estimating 3D hand-object contacts from a single image. The part-level and vertex-level graph-based transformers are cascaded and jointly learned in a coarse-to-fine manner for more accurate contact probabilities. In the second part, we introduce a novel method to diffuse estimated contact states from the hand mesh surface to nearby 3D space and leverage diffused contact probabilities to construct the implicit neural representation for the manipulated object. Benefiting from estimating the interaction patterns between the hand and the object, our method can reconstruct more realistic object meshes, especially for object parts that are in contact with hands. Extensive experiments on challenging benchmarks show that the proposed method outperforms the current state of the arts by a great margin. Our code is publicly available at <a class="link-external link-https" href="https://junxinghu.github.io/projects/hoi.html" rel="external noopener nofollow">this https URL</a>.

Single Depth View Based Real-Time Reconstruction of Hand-Object Interactions

CAMInterHand: Cooperative Attention for Multi-View Interactive Hand Pose and Mesh Reconstruction

In-Hand 3D Object Reconstruction from a Monocular RGB Video

Real-time pose and shape reconstruction of two interacting hands with a single depth camera

Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wild.

Resolving hand‐object occlusion for mixed reality with joint deep learning and model optimization

InteractionFusion

Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Image

Joint Hand-Object 3D Reconstruction From a Single Image With Cross-Branch Feature Fusion

Kinematics-based 3D Human-Object Interaction Reconstruction from Single View

Dynamic Human Body Reconstruction and Motion Tracking with Low-Cost Depth Cameras

Novel-view Synthesis and Pose Estimation for Hand-Object Interaction from Sparse Views

Exploiting Spatial-Temporal Context for Interacting Hand Reconstruction on Monocular RGB Video

Physical Interaction: Reconstructing Hand-object Interactions with Physics

3D Hand Reconstruction via Aggregating Intra and Inter Graphs Guided by Prior Knowledge for Hand-Object Interaction Scenario

Reconstructing Close Human Interactions from Multiple Views

Online Global Non-rigid Registration for 3D Object Reconstruction Using Consumer-level Depth Cameras

Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Images

Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data

RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video