Abstract:Dynamic and dexterous manipulation of objects presents a complex challenge, requiring the synchronization of hand motions with the trajectories of objects to achieve seamless and physically plausible interactions. In this work, we introduce ManiDext, a unified hierarchical diffusion-based framework for generating hand manipulation and grasp poses based on 3D object trajectories. Our key insight is that accurately modeling the contact correspondences between objects and hands during interactions is crucial. Therefore, we propose a continuous correspondence embedding representation that specifies detailed hand correspondences at the vertex level between the object and the hand. This embedding is optimized directly on the hand mesh in a self-supervised manner, with the distance between embeddings reflecting the geodesic distance. Our framework first generates contact maps and correspondence embeddings on the object's surface. Based on these fine-grained correspondences, we introduce a novel approach that integrates the iterative refinement process into the diffusion process during the second stage of hand pose generation. At each step of the denoising process, we incorporate the current hand pose residual as a refinement target into the network, guiding the network to correct inaccurate hand poses. Introducing residuals into each denoising step inherently aligns with traditional optimization process, effectively merging generation and refinement into a single unified framework. Extensive experiments demonstrate that our approach can generate physically plausible and highly realistic motions for various tasks, including single and bimanual hand grasping as well as manipulating both rigid and articulated objects. Code will be available for research purposes.

Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation

Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learning

3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations

ET-SEED: Efficient Trajectory-Level SE(3) Equivariant Diffusion Policy

Diffusion policy: Visuomotor policy learning via action diffusion

Generalizable Humanoid Manipulation with Improved 3D Diffusion Policies

Prediction with Action: Visual Policy Learning via Joint Denoising Process

ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulation

Grasp Diffusion Network: Learning Grasp Generators from Partial Point Clouds with Diffusion Models in SO(3)xR3

DiffGen: Robot Demonstration Generation via Differentiable Physics Simulation, Differentiable Rendering, and Vision-Language Model

Diff-DAgger: Uncertainty Estimation with Diffusion Policy for Robotic Manipulation

ManiDext: Hand-Object Manipulation Synthesis via Continuous Correspondence Embeddings and Residual-Guided Diffusion

SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion

Learning Generalizable 3D Manipulation With 10 Demonstrations

GraspDiff: Grasping Generation for Hand-Object Interaction With Multimodal Guided Diffusion

Robot Shape and Location Retention in Video Generation Using Diffusion Models

Denoising Heat-inspired Diffusion with Insulators for Collision Free Motion Planning

Diff-LfD: Contact-aware Model-based Learning from Visual Demonstration for Robotic Manipulation Via Differentiable Physics-based Simulation and Rendering.

Learning Diffusion Policies from Demonstrations For Compliant Contact-rich Manipulation