Abstract:We present a novel method for populating 3D indoor scenes with virtual humans that can navigate in the environment and interact with objects in a realistic manner. Existing approaches rely on training sequences that contain captured human motions and the 3D scenes they interact with. However, such interaction data are costly, difficult to capture, and can hardly cover all plausible human-scene interactions in complex environments. To address these challenges, we propose a reinforcement learning-based approach that enables virtual humans to navigate in 3D scenes and interact with objects realistically and autonomously, driven by learned motion control policies. The motion control policies employ latent motion action spaces, which correspond to realistic motion primitives and are learned from large-scale motion capture data using a powerful generative motion model. For navigation in a 3D environment, we propose a scene-aware policy with novel state and reward designs for collision avoidance. Combined with navigation mesh-based path-finding algorithms to generate intermediate waypoints, our approach enables the synthesis of diverse human motions navigating in 3D indoor scenes and avoiding obstacles. To generate fine-grained human-object interactions, we carefully curate interaction goal guidance using a marker-based body representation and leverage features based on the signed distance field (SDF) to encode human-scene proximity relations. Our method can synthesize realistic and diverse human-object interactions (e.g.,~sitting on a chair and then getting up) even for out-of-distribution test scenarios with different object shapes, orientations, starting body positions, and poses. Experimental results demonstrate that our approach outperforms state-of-the-art methods in terms of both motion naturalness and diversity. Code and video results are available at: <a class="link-external link-https" href="https://zkf1997.github.io/DIMOS" rel="external noopener nofollow">this https URL</a>.

BIM Sim/3D: Multi-Agent Human Activity Simulation in Indoor Spaces

An agent-based stochastic Occupancy Simulator

A Multi-Purpose Scenario-based Simulator for Smart House Environments

EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environment

A novel approach for occupants' horizontal and vertical movement modeling in non-residential buildings using Immersive Virtual Environment (IVE)

VirtualHome: Simulating Household Activities via Programs

Human-centered In-building Embodied Delivery Benchmark

An agent-based model of public space use

Performance Evaluation of an Agent-Based Occupancy Simulation Model

Privacy-preserving Activity Recognition Using Multimodal Sensors in Smart Office

Simulation of Occupancy in Buildings

SimArch: A Multi-agent System For Human Path Simulation In Architecture Design

An Activity Management System for Office Workers Using Multimodal Data

HumRoboSim: an Autonomous Humanoid Robot Simulation System

Synthesizing Diverse Human Motions in 3D Indoor Scenes

BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments

Building Automation System Data Integration with BIM: Data Structure and Supporting Case Study

Building Occupant Transient Agent-Based Model – Movement Module

Automatic spatial attribute and travel pattern generation for simulating living spaces for elderly individuals living alone

A Virtual Simulation Experimental Platform for Smart Home based on Unity3D

On-demand monitoring of construction projects through a game-like hybrid application of BIM and machine learning