Abstract:In various multimedia applications, it is of great significance to reconstruct 3D meshes of hands and objects from single RGB images. Mesh-based methods mainly resort to mesh displacements by estimating relative positions between hands and objects, while the distance may be inaccurate. Methods based on signed distance function (SDF) learn relative positions by concurrently sampling hand meshes and object meshes; unfortunately, these methods have very limited capability of reconstructing smooth surfaces with rich details. For example, SDF-based methods are inclined to lose the typologies. To the best of our knowledge, only limited works can simultaneously reconstruct the hands and objects with smooth surfaces and accurate relative positions. To this end, we present a novel hybrid model—hand–object Model (HandO) enabling the hand–object 3D reconstruction with smooth surfaces and accurate positions. Critically, our model for the first time makes the hybrid 3D representation for this task by bringing meshes, SDFs, and parametric models together. A feature extractor is employed to extract the image features, and SDF sample points are projected onto these features to extract the local features of each sampled point. Essentially, our model can be naturally extended to reconstruct a whole body holding an object via the new hybrid representation. Additionally, to overcome the lack of training data, a synthetic body-holding dataset is contributed to the community, thus facilitating the research of reconstructing the hand and object. It contains 31763 images of over 50 object categories. Extensive experiments demonstrate that our model can achieve better performance over the competitors on benchmark datasets.

Model Recovery Of Unknown Objects From Discrete Tactile Points

3-D Object Reconstruction Using Tactile Sensing for Multi-fingered Hand

Unknown Objects Grasp Planning Using Force/Torque Measurements

Object shape recognition approach for sparse point clouds from tactile exploration

3D Model Retrieval Based on Spatial and Normal Feature

Sliding Touch-based Exploration for Modeling Unknown Object Shape with Multi-fingered Hands

Haptic Perception Of Unknown Object By Robot Hand: Exploration Strategy And Recognition Approach

Visual-Tactile Sensing for In-Hand Object Reconstruction

Model-Based 3D Contact Geometry Perception for Visual Tactile Sensor

Grasping Unknown Objects Based on 3d Model Reconstruction

3D Dense Reconstruction of Vision-based Tactile Sensor with Coded Markers

PseudoTouch: Efficiently Imaging the Surface Feel of Objects for Robotic Manipulation

Geometric Feature Extraction and Model Reconstruction Based on Scattered Data

Acquisition and Prediction of High-Density Tactile Field Data for Rigid and Flexible Objects

HandO: a Hybrid 3D Hand–object Reconstruction Model for Unknown Objects

AcTExplore: Active Tactile Exploration of Unknown Objects

Unknown object modeling on the basis of vision and pushing manipulation.

Self-supervised 3D Human Mesh Recovery from Noisy Point Clouds

Recent progress on tactile object recognition

3D model database reconstruction based on texture feature points detection

Dynamic Reconstruction of Hand-Object Interaction with Distributed Force-aware Contact Representation