Abstract:We present a novel method for populating 3D indoor scenes with virtual humans that can navigate in the environment and interact with objects in a realistic manner. Existing approaches rely on training sequences that contain captured human motions and the 3D scenes they interact with. However, such interaction data are costly, difficult to capture, and can hardly cover all plausible human-scene interactions in complex environments. To address these challenges, we propose a reinforcement learning-based approach that enables virtual humans to navigate in 3D scenes and interact with objects realistically and autonomously, driven by learned motion control policies. The motion control policies employ latent motion action spaces, which correspond to realistic motion primitives and are learned from large-scale motion capture data using a powerful generative motion model. For navigation in a 3D environment, we propose a scene-aware policy with novel state and reward designs for collision avoidance. Combined with navigation mesh-based path-finding algorithms to generate intermediate waypoints, our approach enables the synthesis of diverse human motions navigating in 3D indoor scenes and avoiding obstacles. To generate fine-grained human-object interactions, we carefully curate interaction goal guidance using a marker-based body representation and leverage features based on the signed distance field (SDF) to encode human-scene proximity relations. Our method can synthesize realistic and diverse human-object interactions (e.g.,~sitting on a chair and then getting up) even for out-of-distribution test scenarios with different object shapes, orientations, starting body positions, and poses. Experimental results demonstrate that our approach outperforms state-of-the-art methods in terms of both motion naturalness and diversity. Code and video results are available at: <a class="link-external link-https" href="https://zkf1997.github.io/DIMOS" rel="external noopener nofollow">this https URL</a>.

Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Control

Learning Hierarchical Behavior and Motion Planning for Autonomous Driving.

Learning Responsive Humanoid Motion Skills from Graph-Powered Motion Matching

Hierarchical Learning Control with Physical Human-Exoskeleton Interaction.

HYPERmotion: Learning Hybrid Behavior Planning for Autonomous Loco-manipulation

Expressive Whole-Body Control for Humanoid Robots

A scalable approach to control diverse behaviors for physically simulated characters

Hierarchical World Models as Visual Whole-Body Humanoid Controllers

Perpetual Humanoid Control for Real-time Simulated Avatars

Hierarchical Robot Learning for Physical Collaboration Between Humans and Robots

Synthesizing Diverse Human Motions in 3D Indoor Scenes

Simulation and Retargeting of Complex Multi-Character Interactions

Hierarchical generative modelling for autonomous robots

Hierarchical Learning Framework for Whole-Body Model Predictive Control of a Real Humanoid Robot

Real-time hierarchical movement animation control for autonomous virtual humans

Hierarchical Planning and Control for Box Loco-Manipulation

Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis

Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion

Learning Hierarchical Control for Robust In-Hand Manipulation

Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots

CHAIRS: Towards Full-Body Articulated Human-Object Interaction