Abstract:We propose a language‐directed virtual human motion generation approach based on musculoskeletal models to achieve interactive and higher‐fidelity virtual human motion. It takes reference motion data, caption and text prompts as inputs, realizing the natural language motion controller through three components: constructing an action‐caption data pool, learning the control policies for imitating the motion, and semantic matching selection. The development of the systems capable of synthesizing natural and life‐like motions for virtual characters has long been a central focus in computer animation. It needs to generate high‐quality motions for characters and provide users with a convenient and flexible interface for guiding character motions. In this work, we propose a language‐directed virtual human motion generation approach based on musculoskeletal models to achieve interactive and higher‐fidelity virtual human motion, which lays the foundation for the development of language‐directed controllers in physics‐based character animation. First, we construct a simplified model of musculoskeletal dynamics for the virtual character. Subsequently, we propose a hierarchical control framework consisting of a trajectory tracking layer and a muscle control layer, obtaining the optimal control policy for imitating the reference motions through the training. We design a multi‐policy aggregation controller based on large language models, which selects the motion policy with the highest similarity to user text commands from the action‐caption data pool, facilitating natural language‐based control of virtual character motions. Experimental results demonstrate that the proposed approach not only generates high‐quality motions highly resembling reference motions but also enables users to effectively guide virtual characters to perform various motions via natural language instructions.

3D Human Motion Synthesis Based on Nonlinear Manifold Learning

Spatio-temporal Manifold Learning for Human Motions via Long-horizon Modeling

MoManifold: Learning to Measure 3D Human Motion via Decoupled Joint Acceleration Manifolds

Learning a Deep Motion Interpolation Network for Human Skeleton Animations

Human motion segmentation using collaborative representations of 3D skeletal sequences.

Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes

Style Synthesis and Editing of Motion Data in Non-Linear Subspace

A Survey on Machine Learning in the Synthesis of Human Motions

3D Skeleton-based Human Motion Prediction with Manifold-Aware GAN

Human Motion Capture Using Wireless Inertial Sensors

Towards 3D Dance Motion Synthesis and Control

Supervised Learning of Motion Style for Real-time Synthesis of 3D Character Animations

Towards Diverse and Natural Scene-aware 3D Human Motion Synthesis

An Interactive Framework for Visually Realistic 3D Motion Synthesis using Evolutionarily-trained Spiking Neural Networks

Generating Continual Human Motion in Diverse 3D Scenes

Synthesizing Moving People with 3D Control

Human Motion Prediction Using Manifold-Aware Wasserstein GAN

A language‐directed virtual human motion generation approach based on musculoskeletal models

Machine Learning Approaches for 3D Motion Synthesis and Musculoskeletal Dynamics Estimation: A Survey

Modeling Trajectories for 3D Motion Analysis

Physics-based Human Motion Estimation and Synthesis from Videos