Full-Body Motion Reconstruction with Sparse Sensing from Graph Perspective

Feiyu Yao,Zongkai Wu,Li Yi

DOI: https://doi.org/10.1609/aaai.v38i7.28483

2024-01-22

Abstract:Estimating 3D full-body pose from sparse sensor data is a pivotal technique employed for the reconstruction of realistic human motions in Augmented Reality and Virtual Reality. However, translating sparse sensor signals into comprehensive human motion remains a challenge since the sparsely distributed sensors in common VR systems fail to capture the motion of full human body. In this paper, we use well-designed Body Pose Graph (BPG) to represent the human body and translate the challenge into a prediction problem of graph missing nodes. Then, we propose a novel full-body motion reconstruction framework based on BPG. To establish BPG, nodes are initially endowed with features extracted from sparse sensor signals. Features from identifiable joint nodes across diverse sensors are amalgamated and processed from both temporal and spatial perspectives. Temporal dynamics are captured using the Temporal Pyramid Structure, while spatial relations in joint movements inform the spatial attributes. The resultant features serve as the foundational elements of the BPG nodes. To further refine the BPG, node features are updated through a graph neural network that incorporates edge reflecting varying joint relations. Our method's effectiveness is evidenced by the attained state-of-the-art performance, particularly in lower body motion, outperforming other baseline methods. Additionally, an ablation study validates the efficacy of each module in our proposed framework.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### The Problem the Paper Attempts to Solve

The paper aims to address the issue of reconstructing full-body motion from sparse sensor data in Virtual Reality (VR) and Augmented Reality (AR) systems. Specifically, common VR systems typically consist of a head-mounted display and handheld controllers, which can provide rich upper body motion information but fail to provide lower body motion data. Due to the sparse distribution of sensors, existing kinematics-based methods face challenges in generating realistic full-body motions.

The paper proposes a new framework that utilizes a carefully designed **Body Pose Graph (BPG)** to represent the human body and transforms the problem into a prediction problem of missing nodes in the graph. By combining a Temporal Pyramid Structure and spatial relationships, joint features are extracted, and node features are updated through a Graph Neural Network (GNN), thereby better capturing the dynamic relationships between joints. This method performs excellently in full-body motion reconstruction tasks, particularly outperforming other baseline methods in the reconstruction of lower body joints.