Abstract:In various multimedia applications, it is of great significance to reconstruct 3D meshes of hands and objects from single RGB images. Mesh-based methods mainly resort to mesh displacements by estimating relative positions between hands and objects, while the distance may be inaccurate. Methods based on signed distance function (SDF) learn relative positions by concurrently sampling hand meshes and object meshes; unfortunately, these methods have very limited capability of reconstructing smooth surfaces with rich details. For example, SDF-based methods are inclined to lose the typologies. To the best of our knowledge, only limited works can simultaneously reconstruct the hands and objects with smooth surfaces and accurate relative positions. To this end, we present a novel hybrid model—hand–object Model (HandO) enabling the hand–object 3D reconstruction with smooth surfaces and accurate positions. Critically, our model for the first time makes the hybrid 3D representation for this task by bringing meshes, SDFs, and parametric models together. A feature extractor is employed to extract the image features, and SDF sample points are projected onto these features to extract the local features of each sampled point. Essentially, our model can be naturally extended to reconstruct a whole body holding an object via the new hybrid representation. Additionally, to overcome the lack of training data, a synthetic body-holding dataset is contributed to the community, thus facilitating the research of reconstructing the hand and object. It contains 31763 images of over 50 object categories. Extensive experiments demonstrate that our model can achieve better performance over the competitors on benchmark datasets.

In-Hand 3D Object Reconstruction from a Monocular RGB Video

Reconstructing Hand-Held Objects from Monocular Video.

CAMInterHand: Cooperative Attention for Multi-View Interactive Hand Pose and Mesh Reconstruction

Reconstructing Hand-Held Objects in 3D from Images and Videos

Learning Hand-Held Object Reconstruction from In-The-Wild Videos

HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video

Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Images

MOHO: Learning Single-view Hand-held Object Reconstruction with Multi-view Occlusion-Aware Supervision

UniHOI: Learning Fast, Dense and Generalizable 4D Reconstruction for Egocentric Hand Object Interaction Videos

3D Reconstruction of Objects in Hands without Real World 3D Supervision

Stereo Hand-Object Reconstruction for Human-to-Robot Handover

HandGCAT: Occlusion-Robust 3D Hand Mesh Reconstruction from Monocular Images.

Robust 3D Reconstruction with an RGB-D Camera

3D Hand Mesh Recovery from Monocular RGB in Camera Space

3DFS: Deformable Dense Depth Fusion and Segmentation for Object Reconstruction from a Handheld Camera

Kinematics-based 3D Human-Object Interaction Reconstruction from Single View

Online Global Non-rigid Registration for 3D Object Reconstruction Using Consumer-level Depth Cameras

HandNeRF: Learning to Reconstruct Hand-Object Interaction Scene from a Single RGB Image

EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild

HandO: a Hybrid 3D Hand–object Reconstruction Model for Unknown Objects

HandOS: 3D Hand Reconstruction in One Stage