Abstract:Human motion prediction is to predict future human states based on the observed human states. However, current research ignores the semantic correlations between body parts (joints and bones) in the observed human states and motion time; thus, the prediction accuracy is limited. To address this issue, we propose a novel semantic correlation attention-based multiorder multiscale feature fusion network (SCAFF), which includes an encoder and a decoder. In the encoder, a multiorder difference calculation module (MODC) is designed to calculate the multiorder difference information of joint and bone attributes in the observed human states. Then, multiple semantic correlation attention-based graph calculation operators (SCA-GCOs) are stacked to extract the multiscale features of the multiorder difference information. Each SCA-GCO captures joint and bone dependencies of the multiorder difference information, refines them with a semantic correlation attention module (SCAM), and captures temporal dynamics of the refined joint and bone dependencies as the output features. Note that SCAM learns a semantic attention mask describing the semantic correlations between body parts and motion time for feature refinement. Afterward, multiple multiorder feature fusion modules (MOFFs) and multiscale feature fusion modules (MSFFs) are designed to fuse the multiscale features of the multiorder difference information extracted by multiple SCA-GCOs, thus obtaining the motion features of the observed human states. Based on the obtained motion features, the decoder recurrently recruits a composite gated recurrent module (CGRM) and multilayer perceptrons (MLPs) to predict future human states. As far as we know, this is the first attempt to consider the semantic correlations between body parts and motion time in human motion prediction. The results on public datasets demonstrate that SCAFF outperforms existing models.

KSOF: Leveraging kinematics and spatio-temporal optimal fusion for human motion prediction

Forecasting Distillation: Enhancing 3D Human Motion Prediction with Guidance Regularization

A Spatio-Temporal Transformer Network for Human Motion Prediction in Human-Robot Collaboration

MFOGCN: Multi-Feature-based Orthogonal Graph Convolutional Network for 3D Human Motion Prediction

TrajectoryCNN: A New Spatio-Temporal Feature Learning Network for Human Motion Prediction

Multiscale Spatial and Temporal Learning for Human Motion Prediction

KD-Former: Kinematic and dynamic coupled transformer network for 3D human motion prediction

DMS-GCN: Dynamic Mutiscale Spatiotemporal Graph Convolutional Networks for Human Motion Prediction

Temporal Consistency Two-Stream CNN for Human Motion Prediction

MSTP-Net: Multiscale Spatio-temporal Parallel Networks for Human Motion Prediction

Towards Accurate 3D Human Motion Prediction from Incomplete Observations

Multi-Scale Incremental Modeling for Enhanced Human Motion Prediction in Human-Robot Collaboration

Class-guided Human Motion Prediction Via Multi-Spatial-temporal Supervision

Semantic Correlation Attention-Based Multiorder Multiscale Feature Fusion Network for Human Motion Prediction

AMHGCN: Adaptive multi-level hypergraph convolution network for human motion prediction

Spatial-Temporal Correlation Modeling for Motion Prediction

Spatiotemporal Consistency Learning from Momentum Cues for Human Motion Prediction

STTG-net: a Spatio-temporal Network for Human Motion Prediction Based on Transformer and Graph Convolution Network

Collaborative Multi-Dynamic Pattern Modeling for Human Motion Prediction

Motion Prediction Using Trajectory Cues

Multilevel Joint Association Networks for Diverse Human Motion Prediction