Abstract:In various multimedia applications, it is of great significance to reconstruct 3D meshes of hands and objects from single RGB images. Mesh-based methods mainly resort to mesh displacements by estimating relative positions between hands and objects, while the distance may be inaccurate. Methods based on signed distance function (SDF) learn relative positions by concurrently sampling hand meshes and object meshes; unfortunately, these methods have very limited capability of reconstructing smooth surfaces with rich details. For example, SDF-based methods are inclined to lose the typologies. To the best of our knowledge, only limited works can simultaneously reconstruct the hands and objects with smooth surfaces and accurate relative positions. To this end, we present a novel hybrid model—hand–object Model (HandO) enabling the hand–object 3D reconstruction with smooth surfaces and accurate positions. Critically, our model for the first time makes the hybrid 3D representation for this task by bringing meshes, SDFs, and parametric models together. A feature extractor is employed to extract the image features, and SDF sample points are projected onto these features to extract the local features of each sampled point. Essentially, our model can be naturally extended to reconstruct a whole body holding an object via the new hybrid representation. Additionally, to overcome the lack of training data, a synthetic body-holding dataset is contributed to the community, thus facilitating the research of reconstructing the hand and object. It contains 31763 images of over 50 object categories. Extensive experiments demonstrate that our model can achieve better performance over the competitors on benchmark datasets.

3D Hand Reconstruction with Both Shape and Appearance from an RGB Image.

gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction

In-Hand 3D Object Reconstruction from a Monocular RGB Video

Coarse-to-Fine Implicit Representation Learning for 3D Hand-Object Reconstruction from a Single RGB-D Image

HandFormer: Hand Pose Reconstructing from a Single RGB Image

HandS3C: 3D Hand Mesh Reconstruction with State Space Spatial Channel Attention from RGB images

HandO: a Hybrid 3D Hand–object Reconstruction Model for Unknown Objects

Consistent 3D Hand Reconstruction in Video Via Self-Supervised Learning

Model-based 3D Hand Reconstruction via Self-Supervised Learning

3D Hand Pose and Shape Estimation from Monocular RGB Via Efficient 2D Cues

AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction

RealisticHands: A Hybrid Model for 3D Hand Reconstruction

Interacting Two-Hand 3D Pose and Shape Reconstruction from Single Color Image

3D Hand Pose Estimation and Reconstruction Based on Multi-Feature Fusion

3D Points Splatting for Real-Time Dynamic Hand Reconstruction

Hand-3d-Studio: A New Multi-View System for 3d Hand Reconstruction.

3D Shape Reconstruction from Free-Hand Sketches.

XHand: Real-time Expressive Hand Avatar

High Fidelity 3D Hand Shape Reconstruction via Scalable Graph Frequency Decomposition

3D Hand Reconstruction via Aggregating Intra and Inter Graphs Guided by Prior Knowledge for Hand-Object Interaction Scenario

3D Hand Shape and Pose Estimation from a Single RGB Image