Abstract:In various multimedia applications, it is of great significance to reconstruct 3D meshes of hands and objects from single RGB images. Mesh-based methods mainly resort to mesh displacements by estimating relative positions between hands and objects, while the distance may be inaccurate. Methods based on signed distance function (SDF) learn relative positions by concurrently sampling hand meshes and object meshes; unfortunately, these methods have very limited capability of reconstructing smooth surfaces with rich details. For example, SDF-based methods are inclined to lose the typologies. To the best of our knowledge, only limited works can simultaneously reconstruct the hands and objects with smooth surfaces and accurate relative positions. To this end, we present a novel hybrid model—hand–object Model (HandO) enabling the hand–object 3D reconstruction with smooth surfaces and accurate positions. Critically, our model for the first time makes the hybrid 3D representation for this task by bringing meshes, SDFs, and parametric models together. A feature extractor is employed to extract the image features, and SDF sample points are projected onto these features to extract the local features of each sampled point. Essentially, our model can be naturally extended to reconstruct a whole body holding an object via the new hybrid representation. Additionally, to overcome the lack of training data, a synthetic body-holding dataset is contributed to the community, thus facilitating the research of reconstructing the hand and object. It contains 31763 images of over 50 object categories. Extensive experiments demonstrate that our model can achieve better performance over the competitors on benchmark datasets.

3D hand reconstruction via aggregating intra and inter graphs guided by prior knowledge for hand-object interaction scenario

3D Hand Reconstruction via Aggregating Intra and Inter Graphs Guided by Prior Knowledge for Hand-Object Interaction Scenario

In-Hand 3D Object Reconstruction from a Monocular RGB Video

CAMInterHand: Cooperative Attention for Multi-View Interactive Hand Pose and Mesh Reconstruction

Hand-object reconstruction via interaction-aware graph attention mechanism

Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Image

Interacting Attention Graph for Single Image Two-Hand Reconstruction

Coarse-to-Fine Hand-Object Pose Estimation with Interaction-Aware Graph Convolutional Network

Joint Hand-Object 3D Reconstruction From a Single Image With Cross-Branch Feature Fusion

3D Human Reconstruction from A Single Depth Image

3D Hand Mesh Recovery from Monocular RGB in Camera Space

3D Hand Pose Estimation in the Wild via Graph Refinement under Adversarial Learning

3D hand pose and mesh estimation via a generic Topology-aware Transformer model

ShapeGraFormer: GraFormer-Based Network for Hand-Object Reconstruction from a Single Depth Map

HandO: a Hybrid 3D Hand–object Reconstruction Model for Unknown Objects

HandGCAT: Occlusion-Robust 3D Hand Mesh Reconstruction from Monocular Images.

Coarse-to-fine cascaded 3D hand reconstruction based on SSGC and MHSA

HandS3C: 3D Hand Mesh Reconstruction with State Space Spatial Channel Attention from RGB images

Joint Hand-Object Pose Estimation with Differentiably-Learned Physical Contact Point Analysis

Single Depth View Based Real-Time Reconstruction of Hand-Object Interactions

Overcoming the Trade-off Between Accuracy and Plausibility in 3D Hand Shape Reconstruction