Abstract:Synthesizing natural interactions between virtual humans and their 3D environments is critical for numerous applications, such as computer games and AR/VR experiences. Our goal is to synthesize humans interacting with a given 3D scene controlled by high-level semantic specifications as pairs of action categories and object instances, e.g., "sit on the chair". The key challenge of incorporating interaction semantics into the generation framework is to learn a joint representation that effectively captures heterogeneous information, including human body articulation, 3D object geometry, and the intent of the interaction. To address this challenge, we design a novel transformer-based generative model, in which the articulated 3D human body surface points and 3D objects are jointly encoded in a unified latent space, and the semantics of the interaction between the human and objects are embedded via positional encoding. Furthermore, inspired by the compositional nature of interactions that humans can simultaneously interact with multiple objects, we define interaction semantics as the composition of varying numbers of atomic action-object pairs. Our proposed generative model can naturally incorporate varying numbers of atomic interactions, which enables synthesizing compositional human-scene interactions without requiring composite interaction data. We extend the PROX dataset with interaction semantic labels and scene instance segmentation to evaluate our method and demonstrate that our method can generate realistic human-scene interactions with semantic control. Our perceptual study shows that our synthesized virtual humans can naturally interact with 3D scenes, considerably outperforming existing methods. We name our method COINS, for COmpositional INteraction Synthesis with Semantic Control. Code and data are available at https://github.com/zkf1997/COINS.

SYNTHESIZE COMPOUND BEHAVIOR STATUS

ShapeSynth: Parameterizing Model Collections for Coupled Shape Exploration and Synthesis

BeSimulator: A Large Language Model Powered Text-based Behavior Simulator

Modeling the Synchronization of Multimodal Perceptions as a Basis for the Emergence of Deterministic Behaviors

On the Expressive Power of Behavior Structure

Constrained Behavior Manipulation for Interactive Assembly in a Virtual Environment

Reactive high-level behavior synthesis for an Atlas humanoid robot

A Resourceful Reframing of Behavior Trees

Adaptive Synthetic Characters for Military Training

Controlling Synthetic Characters in Simulations: A Case for Cognitive Architectures and Sigma

Behavior modeling of virtual manufacturing environment

Compositional Human-Scene Interaction Synthesis with Semantic Control

RealBehavior: A Framework for Faithfully Characterizing Foundation Models' Human-like Behavior Mechanisms

Component-Based Modeling and Code Synthesis for Cyclic Programs

Research on Behavior Modeling Technology of Virtual Environment

Habits and Goals in Synergy: a Variational Bayesian Framework for Behavior

Natural Interaction Synthesizing in Virtual Teleconferencing

Scenario-Based Behavioral Nonexistent Consistency Checking for Cyber-Physical Systems

Imperatives for Virtual Humans

Priority-based state machine synthesis that relaxes behavior design of multi-arm manipulators in dynamic environments

Deriving Available Behavior All out from Incompatible Component Compositions