Abstract:We propose a language‐directed virtual human motion generation approach based on musculoskeletal models to achieve interactive and higher‐fidelity virtual human motion. It takes reference motion data, caption and text prompts as inputs, realizing the natural language motion controller through three components: constructing an action‐caption data pool, learning the control policies for imitating the motion, and semantic matching selection. The development of the systems capable of synthesizing natural and life‐like motions for virtual characters has long been a central focus in computer animation. It needs to generate high‐quality motions for characters and provide users with a convenient and flexible interface for guiding character motions. In this work, we propose a language‐directed virtual human motion generation approach based on musculoskeletal models to achieve interactive and higher‐fidelity virtual human motion, which lays the foundation for the development of language‐directed controllers in physics‐based character animation. First, we construct a simplified model of musculoskeletal dynamics for the virtual character. Subsequently, we propose a hierarchical control framework consisting of a trajectory tracking layer and a muscle control layer, obtaining the optimal control policy for imitating the reference motions through the training. We design a multi‐policy aggregation controller based on large language models, which selects the motion policy with the highest similarity to user text commands from the action‐caption data pool, facilitating natural language‐based control of virtual character motions. Experimental results demonstrate that the proposed approach not only generates high‐quality motions highly resembling reference motions but also enables users to effectively guide virtual characters to perform various motions via natural language instructions.

Competitive Motion Synthesis Based on Hybrid Control.

A Motion-based User Interface for the Control of Virtual Humans Performing Sports.

Dynamic Movement Primitive Based Motion Retargeting for Dual-Arm Sign Language Motions

Example-Based Automatic Music-Driven Conventional Dance Motion Synthesis

Real Time Falling Animation with Active and Protective Responses

Fragment-Based Responsive Character Motion for Interactive Games

Responsive Action Generation By Physically-Based Motion Retrieval And Adaptation

Interactive Animation of Virtual Characters: Application to Virtual Kung-Fu Fighting

Simulating Reactive Motions for Motion Capture Animation

Moving Personal Robots in Real-Time Using Primitive Motions

Morphology Independent Motion Retrieval and Control

A language‐directed virtual human motion generation approach based on musculoskeletal models

InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint

Real-time Controllable Motion Transition for Characters.

MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controls

Taming Diffusion Probabilistic Models for Character Control

Interactive Character Control with Auto-Regressive Motion Diffusion Models

Programmable Motion Generation for Open-Set Motion Control Tasks

Towards 3D Dance Motion Synthesis and Control

An Interactive Framework for Visually Realistic 3D Motion Synthesis using Evolutionarily-trained Spiking Neural Networks