Abstract:We present a novel method for populating 3D indoor scenes with virtual humans that can navigate in the environment and interact with objects in a realistic manner. Existing approaches rely on training sequences that contain captured human motions and the 3D scenes they interact with. However, such interaction data are costly, difficult to capture, and can hardly cover all plausible human-scene interactions in complex environments. To address these challenges, we propose a reinforcement learning-based approach that enables virtual humans to navigate in 3D scenes and interact with objects realistically and autonomously, driven by learned motion control policies. The motion control policies employ latent motion action spaces, which correspond to realistic motion primitives and are learned from large-scale motion capture data using a powerful generative motion model. For navigation in a 3D environment, we propose a scene-aware policy with novel state and reward designs for collision avoidance. Combined with navigation mesh-based path-finding algorithms to generate intermediate waypoints, our approach enables the synthesis of diverse human motions navigating in 3D indoor scenes and avoiding obstacles. To generate fine-grained human-object interactions, we carefully curate interaction goal guidance using a marker-based body representation and leverage features based on the signed distance field (SDF) to encode human-scene proximity relations. Our method can synthesize realistic and diverse human-object interactions (e.g.,~sitting on a chair and then getting up) even for out-of-distribution test scenarios with different object shapes, orientations, starting body positions, and poses. Experimental results demonstrate that our approach outperforms state-of-the-art methods in terms of both motion naturalness and diversity. Code and video results are available at: <a class="link-external link-https" href="https://zkf1997.github.io/DIMOS" rel="external noopener nofollow">this https URL</a>.

PlaMo: Plan and Move in Rich 3D Physical Environments

PLATO: Planning with LLMs and Affordances for Tool Manipulation

Responsive Action Generation By Physically-Based Motion Retrieval And Adaptation

SIMS: Simulating Human-Scene Interactions with Real World Script Planning

PACE: Data-Driven Virtual Agent Interaction in Dense and Cluttered Environments

Synthesizing Diverse Human Motions in 3D Indoor Scenes

SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World

Hierarchical Planning and Control for Box Loco-Manipulation

Reactive Task and Motion Planning for Robust Whole-Body Dynamic Locomotion in Constrained Environments

HYPERmotion: Learning Hybrid Behavior Planning for Autonomous Loco-manipulation

E$ \mathbf{^3} $MoP: Efficient Motion Planning Based on Heuristic-Guided Motion Primitives Pruning and Path Optimization With Sparse-Banded Structure

Coordinate Invariant User-Guided Constrained Path Planning with Reactive Rapidly Expanding Plane-Oriented Escaping Trees

Planning for Complex Non-prehensile Manipulation Among Movable Objects by Interleaving Multi-Agent Pathfinding and Physics-Based Simulation

PhyPlan: Generalizable and Rapid Physical Task Planning with Physics Informed Skill Networks for Robot Manipulators

Hierarchical Adaptive Motion Planning with Nonlinear Model Predictive Control for Safety-Critical Collaborative Loco-Manipulation

Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance

Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environments

Simultaneous Scene Reconstruction and Whole-Body Motion Planning for Safe Operation in Dynamic Environments

SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning

The Wanderings of Odysseus in 3D Scenes

Motion Planning for a Humanoid Walking in a 3D Space