Abstract:Natural language plays a critical role in many computer vision applications, such as image captioning, visual question answering, and cross-modal retrieval, to provide fine-grained semantic information. Unfortunately, while human pose is key to human understanding, current 3D human pose datasets lack detailed language descriptions. To address this issue, we have introduced the PoseScript dataset. This dataset pairs more than six thousand 3D human poses from AMASS with rich human-annotated descriptions of the body parts and their spatial relationships. Additionally, to increase the size of the dataset to a scale that is compatible with data-hungry learning algorithms, we have proposed an elaborate captioning process that generates automatic synthetic descriptions in natural language from given 3D keypoints. This process extracts low-level pose information, known as "posecodes", using a set of simple but generic rules on the 3D keypoints. These posecodes are then combined into higher level textual descriptions using syntactic rules. With automatic annotations, the amount of available data significantly scales up (100k), making it possible to effectively pretrain deep models for finetuning on human captions. To showcase the potential of annotated poses, we present three multi-modal learning tasks that utilize the PoseScript dataset. Firstly, we develop a pipeline that maps 3D poses and textual descriptions into a joint embedding space, allowing for cross-modal retrieval of relevant poses from large-scale datasets. Secondly, we establish a baseline for a text-conditioned model generating 3D poses. Thirdly, we present a learned process for generating pose descriptions. These applications demonstrate the versatility and usefulness of annotated poses in various tasks and pave the way for future research in the field.

MotionScript: Natural Language Descriptions for Expressive 3D Human Motions

A Script Engine for Realistic Human Motion Generation

Contact-aware Human Motion Generation from Textual Descriptions

Motion Generation from Fine-grained Textual Descriptions

Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset

PoseScript: Linking 3D Human Poses and Natural Language

Text-to-Motion Retrieval: Towards Joint Understanding of Human Motion Data and Natural Language

Generating Human Motion in 3D Scenes from Text Descriptions

Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance

SIMS: Simulating Human-Scene Interactions with Real World Script Planning

AMD:Anatomical Motion Diffusion with Interpretable Motion Decomposition and Fusion

Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs

MotionGPT: Human Motion Synthesis with Improved Diversity and Realism via GPT-3 Prompting

MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding

DiverseMotion: Towards Diverse Human Motion Generation Via Discrete Diffusion

Unimotion: Unifying 3D Human Motion Synthesis and Understanding

FreeMotion: MoCap-Free Human Motion Synthesis with Multimodal Large Language Models

The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion

Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptions

Generating Holistic 3D Human Motion from Speech