Abstract:Accurate hand motion capture and standardized 3D representation are essential for various hand-related tasks. Collecting keypoints-only data, while efficient and cost-effective, results in low-fidelity representations and lacks surface information. Furthermore, data inconsistencies across sources challenge their integration and use. We present UniHands, a novel method for creating standardized yet personalized hand models from wild-collected keypoints from diverse sources. Unlike existing neural implicit representation methods, UniHands uses the widely-adopted parametric models MANO and NIMBLE, providing a more scalable and versatile solution. It also derives unified hand joints from the meshes, which facilitates seamless integration into various hand-related tasks. Experiments on the FreiHAND and InterHand2.6M datasets demonstrate its ability to precisely reconstruct hand mesh vertices and keypoints, effectively capturing high-degree articulation motions. Empirical studies involving nine participants show a clear preference for our unified joints over existing configurations for accuracy and naturalism (p-value 0.016).

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to create standardized and personalized 3D hand models from low - quality and inconsistent hand - key - point data collected from various sources. Specifically, existing hand - motion - capture methods either lack surface information (such as skeleton - tracking based only on key - points), or are costly and time - consuming (such as mesh - based capture). In addition, data from different sources vary greatly in format and quality, making it difficult to integrate and use. To solve these problems, the authors propose the **UniHands** method, which is able to: 1. **Unify key - point data from multiple sources**: Standardize hand - key - point data from different sources to generate high - quality 3D hand models. 2. **Reconstruct hand models in a personalized way**: Utilize widely - adopted parameterized hand models (such as MANO and NIMBLE) to reconstruct personalized hand models according to individual characteristics. 3. **Extract unified hand joints from the reconstructed mesh**: Combine the advantages of MANO and NIMBLE models to generate more accurate and natural hand joints for better application in various hand - related tasks. ### Specific Problem Description - **Low - quality representation**: Data based only on key - points is efficient and low - cost, but cannot provide sufficient surface information, resulting in low representation accuracy. - **Data inconsistency**: Data from different sources vary greatly in format and quality, making it difficult to integrate and use. - **Limitations of existing methods**: Existing neural implicit representation methods are difficult to extend and apply to diverse scenarios. ### Solution UniHands solves the above problems through the following steps: 1. **Coarse - to - fine optimization**: Use the gradient - descent method to optimize the pose (\(\theta\)), shape (\(\beta\)) and global wrist rotation (\(R_w\)) of the MANO hand model to align the input key - points and the reconstructed mesh vertices. - Coarse stage: Initial pose and average shape, optimize wrist rotation. - Fine stage: Further optimize pose and shape parameters and fine - tune wrist rotation. 2. **Extraction of unified joints**: By combining the advantages of MANO and NIMBLE models, extract more accurate and natural hand joints from the reconstructed mesh. - Align MANO and NIMBLE meshes and train a multi - layer perceptron (MLP) to directly predict NIMBLE - style joints. ### Experimental Verification - **Mesh - reconstruction evaluation**: Experiments were carried out on the FreiHAND and InterHand2.6M datasets, and the results showed that the reconstruction error was very small (PJ/PV < 0.1mm), indicating the effectiveness of the method. - **Joint evaluation**: Through a user study, the performance of the hand joints of the MANO, NIMBLE and UniHands fusion models in terms of realism, accuracy and naturalness was compared, and the results showed that the fusion model was significantly better than the other two models (p - value = 0.016). In conclusion, UniHands provides an efficient and high - quality hand - reconstruction method, which is suitable for application scenarios such as XR hand - tracking, animation and hand - synthesis.

UniHands: Unifying Various Wild-Collected Keypoints for Personalized Hand Reconstruction

CAMInterHand: Cooperative Attention for Multi-View Interactive Hand Pose and Mesh Reconstruction

Personalized Hand Modeling from Multiple Postures with Multi‐View Color Images

3D Hand Reconstruction via Aggregating Intra and Inter Graphs Guided by Prior Knowledge for Hand-Object Interaction Scenario

XHand: Real-time Expressive Hand Avatar

HandOS: 3D Hand Reconstruction in One Stage

OmniHands: Towards Robust 4D Hand Mesh Recovery via A Versatile Transformer

Overcoming the Trade-off Between Accuracy and Plausibility in 3D Hand Shape Reconstruction

Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimation

Im2Hands: Learning Attentive Implicit Representation of Interacting Two-Hand Shapes

Progressively Global-Local Fusion with Explicit Guidance for Accurate and Robust 3d Hand Pose Reconstruction

HandO: a Hybrid 3D Hand–object Reconstruction Model for Unknown Objects

HandNeRF: Neural Radiance Fields for Animatable Interacting Hands

Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Image

Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering

RealisticHands: A Hybrid Model for 3D Hand Reconstruction

Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wild.

UniGrasp: Learning a Unified Model to Grasp with Multifingered Robotic Hands

Two Hands Are Better Than One: Resolving Hand to Hand Intersections via Occupancy Networks

Biomimetic Evaluation of an Underwater Soft Hand Through Deep Learning-based 3D Pose Reconstruction