Abstract:Data-driven modeling of human motions is ubiquitous in computer graphics and computer vision applications, such as synthesizing realistic motions or recognizing actions. Recent research has shown that such problems can be approached by learning a natural motion manifold using deep learning to address the shortcomings of traditional data-driven approaches. However, previous methods can be sub-optimal for two reasons. First, the skeletal information has not been fully utilized for feature extraction. Unlike images, it is difficult to define spatial proximity in skeletal motions in the way that deep networks can be applied. Second, motion is time-series data with strong multi-modal temporal correlations. A frame could be followed by several candidate frames leading to different motions; long-range dependencies exist where a number of frames in the beginning correlate to a number of frames later. Ineffective modeling would either under-estimate the multi-modality and variance, resulting in featureless mean motion or over-estimate them resulting in jittery motions. In this paper, we propose a new deep network to tackle these challenges by creating a natural motion manifold that is versatile for many applications. The network has a new spatial component for feature extraction. It is also equipped with a new batch prediction model that predicts a large number of frames at once, such that long-term temporally-based objective functions can be employed to correctly learn the motion multi-modality and variances. With our system, long-duration motions can be predicted/synthesized using an open-loop setup where the motion retains the dynamics accurately. It can also be used for denoising corrupted motions and synthesizing new motions with given control signals. We demonstrate that our system can create superior results comparing to existing work in multiple applications.

Motion Prediction With Gaussian Process Dynamical Models

GaussianPrediction: Dynamic 3D Gaussian Prediction for Motion Extrapolation and Free View Synthesis

Forecasting Distillation: Enhancing 3D Human Motion Prediction with Guidance Regularization

Probabilistic trajectory prediction with Gaussian mixture models

Human Motion Prediction Using Manifold-Aware Wasserstein GAN

Physics-Informed Spatio-Temporal Model for Human Mobility Prediction

Prediction with Approximated Gaussian Process Dynamical Models

Spatial–temporal modeling for prediction of stylized human motion

State-space Decomposition Model for Video Prediction Considering Long-term Motion Trend

A Data-Efficient Approach for Long-Term Human Motion Prediction Using Maps of Dynamics

Learning Semantic Latent Directions for Accurate and Controllable Human Motion Prediction

MotionWavelet: Human Motion Prediction via Wavelet Manifold Learning

Towards Generalizable and Interpretable Motion Prediction: A Deep Variational Bayes Approach

Probabilistic Long-term Vehicle Trajectory Prediction Via Driver Awareness Model

Human Motion Prediction under Unexpected Perturbation

Human Joint Kinematics Diffusion-Refinement for Stochastic Motion Prediction

Motion Graph Unleashed: A Novel Approach to Video Prediction

Spatio-temporal Manifold Learning for Human Motions via Long-horizon Modeling

Stochastic Trajectory Prediction Via Motion Indeterminacy Diffusion

MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions

Enhanced Human-Robot Collaboration using Constrained Probabilistic Human-Motion Prediction