Abstract:We present a novel method for populating 3D indoor scenes with virtual humans that can navigate in the environment and interact with objects in a realistic manner. Existing approaches rely on training sequences that contain captured human motions and the 3D scenes they interact with. However, such interaction data are costly, difficult to capture, and can hardly cover all plausible human-scene interactions in complex environments. To address these challenges, we propose a reinforcement learning-based approach that enables virtual humans to navigate in 3D scenes and interact with objects realistically and autonomously, driven by learned motion control policies. The motion control policies employ latent motion action spaces, which correspond to realistic motion primitives and are learned from large-scale motion capture data using a powerful generative motion model. For navigation in a 3D environment, we propose a scene-aware policy with novel state and reward designs for collision avoidance. Combined with navigation mesh-based path-finding algorithms to generate intermediate waypoints, our approach enables the synthesis of diverse human motions navigating in 3D indoor scenes and avoiding obstacles. To generate fine-grained human-object interactions, we carefully curate interaction goal guidance using a marker-based body representation and leverage features based on the signed distance field (SDF) to encode human-scene proximity relations. Our method can synthesize realistic and diverse human-object interactions (e.g.,~sitting on a chair and then getting up) even for out-of-distribution test scenarios with different object shapes, orientations, starting body positions, and poses. Experimental results demonstrate that our approach outperforms state-of-the-art methods in terms of both motion naturalness and diversity. Code and video results are available at: <a class="link-external link-https" href="https://zkf1997.github.io/DIMOS" rel="external noopener nofollow">this https URL</a>.

Composable Semi parametric Modelling for Long range Motion Generation

CoMA: Compositional Human Motion Generation with Multi-modal Agents

Programmable Motion Generation for Open-Set Motion Control Tasks

3D Human Motion Synthesis Based on Nonlinear Manifold Learning

CoMo: Controllable Motion Generation through Language Guided Pose Code Editing

Generating Continual Human Motion in Diverse 3D Scenes

Synthesizing Long-Term Human Motions with Diffusion Models via Coherent Sampling

Task-Oriented Human-Object Interactions Generation with Implicit Neural Representations

Towards Efficient and Diverse Generative Model for Unconditional Human Motion Synthesis

FreeMotion: MoCap-Free Human Motion Synthesis with Multimodal Large Language Models

Sitcom-Crafter: A Plot-Driven Human Motion Generation System in 3D Scenes

Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models

The Wanderings of Odysseus in 3D Scenes

Multi-Resolution Generative Modeling of Human Motion from Limited Data

Spatio-temporal Manifold Learning for Human Motions via Long-horizon Modeling

Human Motion Generation with Multifactor Models

Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony

Synthesizing Diverse Human Motions in 3D Indoor Scenes

Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture

Everything2Motion: Synchronizing Diverse Inputs Via a Unified Framework for Human Motion Synthesis

Dynamic Future Net: Diversified Human Motion Generation