Abstract:In various multimedia applications, it is of great significance to reconstruct 3D meshes of hands and objects from single RGB images. Mesh-based methods mainly resort to mesh displacements by estimating relative positions between hands and objects, while the distance may be inaccurate. Methods based on signed distance function (SDF) learn relative positions by concurrently sampling hand meshes and object meshes; unfortunately, these methods have very limited capability of reconstructing smooth surfaces with rich details. For example, SDF-based methods are inclined to lose the typologies. To the best of our knowledge, only limited works can simultaneously reconstruct the hands and objects with smooth surfaces and accurate relative positions. To this end, we present a novel hybrid model—hand–object Model (HandO) enabling the hand–object 3D reconstruction with smooth surfaces and accurate positions. Critically, our model for the first time makes the hybrid 3D representation for this task by bringing meshes, SDFs, and parametric models together. A feature extractor is employed to extract the image features, and SDF sample points are projected onto these features to extract the local features of each sampled point. Essentially, our model can be naturally extended to reconstruct a whole body holding an object via the new hybrid representation. Additionally, to overcome the lack of training data, a synthetic body-holding dataset is contributed to the community, thus facilitating the research of reconstructing the hand and object. It contains 31763 images of over 50 object categories. Extensive experiments demonstrate that our model can achieve better performance over the competitors on benchmark datasets.

gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction

AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction

In-Hand 3D Object Reconstruction from a Monocular RGB Video

3D Hand Reconstruction with Both Shape and Appearance from an RGB Image.

HOISDF: Constraining 3D Hand-Object Pose Estimation with Global Signed Distance Fields

DDF-HO: Hand-Held Object Reconstruction via Conditional Directed Distance Field

SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images

Coarse-to-Fine Implicit Representation Learning for 3D Hand-Object Reconstruction from a Single RGB-D Image

PoseSDF: Simultaneous 3D Human Shape Reconstruction and Gait Pose Estimation Using Signed Distance Functions

SeSDF: Self-evolved Signed Distance Field for Implicit 3D Clothed Human Reconstruction

TouchSDF: A DeepSDF Approach for 3D Shape Reconstruction using Vision-Based Tactile Sensing

Robust 3D Reconstruction with an RGB-D Camera

HandO: a Hybrid 3D Hand–object Reconstruction Model for Unknown Objects

SMR: Spatial-Guided Model-Based Regression for 3D Hand Pose and Mesh Reconstruction

GSDF: 3DGS Meets SDF for Improved Rendering and Reconstruction

The Geometry Enhanced Deep Implicit Function Based 3D Reconstruction for Objects in a Real-Scene Image

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation

A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation

Hand-3d-Studio: A New Multi-View System for 3d Hand Reconstruction.

S3DS: Self-supervised Learning of 3D Skeletons from Single View Images

CrossSDF: 3D Reconstruction of Thin Structures From Cross-Sections