Abstract:We propose a language‐directed virtual human motion generation approach based on musculoskeletal models to achieve interactive and higher‐fidelity virtual human motion. It takes reference motion data, caption and text prompts as inputs, realizing the natural language motion controller through three components: constructing an action‐caption data pool, learning the control policies for imitating the motion, and semantic matching selection. The development of the systems capable of synthesizing natural and life‐like motions for virtual characters has long been a central focus in computer animation. It needs to generate high‐quality motions for characters and provide users with a convenient and flexible interface for guiding character motions. In this work, we propose a language‐directed virtual human motion generation approach based on musculoskeletal models to achieve interactive and higher‐fidelity virtual human motion, which lays the foundation for the development of language‐directed controllers in physics‐based character animation. First, we construct a simplified model of musculoskeletal dynamics for the virtual character. Subsequently, we propose a hierarchical control framework consisting of a trajectory tracking layer and a muscle control layer, obtaining the optimal control policy for imitating the reference motions through the training. We design a multi‐policy aggregation controller based on large language models, which selects the motion policy with the highest similarity to user text commands from the action‐caption data pool, facilitating natural language‐based control of virtual character motions. Experimental results demonstrate that the proposed approach not only generates high‐quality motions highly resembling reference motions but also enables users to effectively guide virtual characters to perform various motions via natural language instructions.

Towards Generating Virtual Movement from Textual Instructions A Case Study in Quality Assessment

Contact-aware Human Motion Generation from Textual Descriptions

A Motion-based User Interface for the Control of Virtual Humans Performing Sports.

Nonlinear Methods to Quantify Movement Variability in Human-Humanoid Interaction Activities

Motion Generation from Fine-grained Textual Descriptions

Quantification of Human Movement for Assessment in Automated Exercise Coaching

Recognition and Scoring Physical Exercises via Temporal and Relative Analysis of Skeleton Nodes Extracted from the Kinect Sensor

Qualitative assessment of recurrent human motion

Motion Capture Benchmark of Real Industrial Tasks and Traditional Crafts for Human Movement Analysis

T2M-X: Learning Expressive Text-to-Motion Generation from Partially Annotated Data

Generating Human Motion in 3D Scenes from Text Descriptions

Responsive Action Generation By Physically-Based Motion Retrieval And Adaptation

HumanTOMATO: Text-aligned Whole-body Motion Generation

TEMOS: Generating diverse human motions from textual descriptions

A language‐directed virtual human motion generation approach based on musculoskeletal models

Automatic Error Analysis of Human Motor Performance for Interactive Coaching in Virtual Reality

Assessing and improving human movements using sensitivity analysis and digital human simulation

Measuring the Quality of Exercises

Enhancing Motion Variation in Text-to-Motion Models via Pose and Video Conditioned Editing

A-MAL: Automatic Movement Assessment Learning from Properly Performed Movements in 3D Skeleton Videos

Learning Human Body Motions from Skeleton-Based Observations for Robot-Assisted Therapy