Abstract:We propose a language‐directed virtual human motion generation approach based on musculoskeletal models to achieve interactive and higher‐fidelity virtual human motion. It takes reference motion data, caption and text prompts as inputs, realizing the natural language motion controller through three components: constructing an action‐caption data pool, learning the control policies for imitating the motion, and semantic matching selection. The development of the systems capable of synthesizing natural and life‐like motions for virtual characters has long been a central focus in computer animation. It needs to generate high‐quality motions for characters and provide users with a convenient and flexible interface for guiding character motions. In this work, we propose a language‐directed virtual human motion generation approach based on musculoskeletal models to achieve interactive and higher‐fidelity virtual human motion, which lays the foundation for the development of language‐directed controllers in physics‐based character animation. First, we construct a simplified model of musculoskeletal dynamics for the virtual character. Subsequently, we propose a hierarchical control framework consisting of a trajectory tracking layer and a muscle control layer, obtaining the optimal control policy for imitating the reference motions through the training. We design a multi‐policy aggregation controller based on large language models, which selects the motion policy with the highest similarity to user text commands from the action‐caption data pool, facilitating natural language‐based control of virtual character motions. Experimental results demonstrate that the proposed approach not only generates high‐quality motions highly resembling reference motions but also enables users to effectively guide virtual characters to perform various motions via natural language instructions.

Pose Synthesis of Virtual Character Based on Statistical Learning

Inverse kinematics based on statistical learning

Gaussian Process Latent Variable Models for Inverse Kinematics

Motion Editing for Quadruped Characters Via Latent Frequency Embedding

Interactive Character Posing by Sparse Coding

Learning-based pose edition for efficient and interactive design

A language‐directed virtual human motion generation approach based on musculoskeletal models

Whole Body Posture Generation of Virtual Characters by Combining Posture Examples and Prioritized Inverse Kinematics

Pose Representations for Deep Skeletal Animation

A constrained inverse kinematics technique for real-time motion capture animation

Human Motion Capture Using Wireless Inertial Sensors

Interactive Animation of Virtual Characters: Application to Virtual Kung-Fu Fighting

Inverse kinematics using dynamic joint parameters: inverse kinematics animation synthesis learnt from sub-divided motion micro-segments

A Method for Character Animation Generation and Control

Pose-to-Motion: Cross-Domain Motion Retargeting with Pose Prior

Optimization algorithm of an artificial neural network-based controller and simulation method for animated virtual idol characters

Real-Time Synthesis of 3d Animations by Learning Parametric Gaussians Using Self-Organizing Mixture Networks

An Interactive Framework for Visually Realistic 3D Motion Synthesis using Evolutionarily-trained Spiking Neural Networks

SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data

Pose‐to‐Motion: Cross‐Domain Motion Retargeting with Pose Prior

PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling