Abstract:Objective: Marker-based motion capture, considered the gold standard in human motion analysis, is expensive and requires trained personnel. Advances in inertial sensing and computer vision offer new opportunities to obtain research-grade assessments in clinics and natural environments. A challenge that discourages clinical adoption, however, is the need for careful sensor-to-body alignment, which slows the data collection process in clinics and is prone to errors when patients take the sensors home. Methods: We propose deep learning models to estimate human movement with noisy data from videos (VideoNet), inertial sensors (IMUNet), and a combination of the two (FusionNet), obviating the need for careful calibration. The video and inertial sensing data used to train the models were generated synthetically from a marker-based motion capture dataset of a broad range of activities and augmented to account for sensor-misplacement and camera-occlusion errors. The models were tested using real data that included walking, jogging, squatting, sit-to-stand, and other activities. Results: On calibrated data, IMUNet was as accurate as state-of-the-art models, while VideoNet and FusionNet reduced mean ± std root-mean-squared errors by 7.6 ± 5.4 ° and 5.9 ± 3.3 °, respectively. Importantly, all the newly proposed models were less sensitive to noise than existing approaches, reducing errors by up to 14.0 ± 5.3 ° for sensor-misplacement errors of up to 30.0 ± 13.7 ° and by up to 7.4 ± 5.5 ° for joint-center-estimation errors of up to 101.1 ± 11.2 mm, across joints. Conclusion: These tools offer clinicians and patients the opportunity to estimate movement with research-grade accuracy, without the need for time-consuming calibration steps or the high costs associated with commercial products such as Theia3D or Xsens, helping democratize the diagnosis, prognosis, and treatment of neuromusculoskeletal conditions.

Physical Non-inertial Poser (PNP): Modeling Non-inertial Effects in Sparse-inertial Human Motion Capture

Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors

Fast Human Motion reconstruction from sparse inertial measurement units considering the human shape

Human Motion Capture Using Wireless Inertial Sensors

Real-time Physics-based Motion Capture with Sparse Sensors

Ultra Inertial Poser: Scalable Motion Capture and Tracking from Sparse Inertial Sensors and Ultra-Wideband Ranging

Dynamic Inertial Poser (DynaIP): Part-Based Motion Dynamics Learning for Enhanced Human Pose Estimation with Sparse Inertial Sensors

Real-Time Human Motion Capture Based on Wearable Inertial Sensor Networks

Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture

Markerless Motion Tracking With Noisy Video and IMU Data

A Scalable and Wearable Self-Sensing IMU Sensor Network for Personalized Human Motion and Deformation Capture

Predicting Missing Markers In Human Motion Capture Using L1-Sparse Representation

Physics-Guided Human Motion Capture with Pose Probability Modeling

SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data

Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videos

3D Deformation Capture Via a Configurable Self-Sensing IMU Sensor Network

Reconstructing 3D human pose and shape from a single image and sparse IMUs

Transformer Inertial Poser: Real-time Human Motion Reconstruction from Sparse IMUs with Simultaneous Terrain Generation

Fusion Poser: 3D Human Pose Estimation Using Sparse IMUs and Head Trackers in Real Time

DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion