Abstract:We propose a language‐directed virtual human motion generation approach based on musculoskeletal models to achieve interactive and higher‐fidelity virtual human motion. It takes reference motion data, caption and text prompts as inputs, realizing the natural language motion controller through three components: constructing an action‐caption data pool, learning the control policies for imitating the motion, and semantic matching selection. The development of the systems capable of synthesizing natural and life‐like motions for virtual characters has long been a central focus in computer animation. It needs to generate high‐quality motions for characters and provide users with a convenient and flexible interface for guiding character motions. In this work, we propose a language‐directed virtual human motion generation approach based on musculoskeletal models to achieve interactive and higher‐fidelity virtual human motion, which lays the foundation for the development of language‐directed controllers in physics‐based character animation. First, we construct a simplified model of musculoskeletal dynamics for the virtual character. Subsequently, we propose a hierarchical control framework consisting of a trajectory tracking layer and a muscle control layer, obtaining the optimal control policy for imitating the reference motions through the training. We design a multi‐policy aggregation controller based on large language models, which selects the motion policy with the highest similarity to user text commands from the action‐caption data pool, facilitating natural language‐based control of virtual character motions. Experimental results demonstrate that the proposed approach not only generates high‐quality motions highly resembling reference motions but also enables users to effectively guide virtual characters to perform various motions via natural language instructions.

A Script Engine for Realistic Human Motion Generation

Automatic Generation of Human Animation Based on Motion Programming.

Automatic Generation of Human Animation Based on Motion Programming: Virtual Humans and Social Agents

HumRoboSim: an Autonomous Humanoid Robot Simulation System

Sketch Based Multi-source 3D Animation Transfer

Sitcom-Crafter: A Plot-Driven Human Motion Generation System in 3D Scenes

Everything2Motion: Synchronizing Diverse Inputs Via a Unified Framework for Human Motion Synthesis

MotionScript: Natural Language Descriptions for Expressive 3D Human Motions

Real Time Falling Animation with Active and Protective Responses

SIMS: Simulating Human-Scene Interactions with Real World Script Planning

Morph: A Motion-free Physics Optimization Framework for Human Motion Generation

A language‐directed virtual human motion generation approach based on musculoskeletal models

A group of novel approaches and a toolkit for motion capture data reusing

Simulating Reactive Motions for Motion Capture Animation

Interactive Animation of Virtual Characters: Application to Virtual Kung-Fu Fighting

Implementation of a notation-based motion choreography system

Video-based Characters

FreeMotion: MoCap-Free Human Motion Synthesis with Multimodal Large Language Models

Towards Developing an Easy-To-Use Scripting Environment for Animating Virtual Characters

Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs

An Interactive Framework for Visually Realistic 3D Motion Synthesis using Evolutionarily-trained Spiking Neural Networks