Abstract:Objective: Marker-based motion capture, considered the gold standard in human motion analysis, is expensive and requires trained personnel. Advances in inertial sensing and computer vision offer new opportunities to obtain research-grade assessments in clinics and natural environments. A challenge that discourages clinical adoption, however, is the need for careful sensor-to-body alignment, which slows the data collection process in clinics and is prone to errors when patients take the sensors home. Methods: We propose deep learning models to estimate human movement with noisy data from videos (VideoNet), inertial sensors (IMUNet), and a combination of the two (FusionNet), obviating the need for careful calibration. The video and inertial sensing data used to train the models were generated synthetically from a marker-based motion capture dataset of a broad range of activities and augmented to account for sensor-misplacement and camera-occlusion errors. The models were tested using real data that included walking, jogging, squatting, sit-to-stand, and other activities. Results: On calibrated data, IMUNet was as accurate as state-of-the-art models, while VideoNet and FusionNet reduced mean ± std root-mean-squared errors by 7.6 ± 5.4 ° and 5.9 ± 3.3 °, respectively. Importantly, all the newly proposed models were less sensitive to noise than existing approaches, reducing errors by up to 14.0 ± 5.3 ° for sensor-misplacement errors of up to 30.0 ± 13.7 ° and by up to 7.4 ± 5.5 ° for joint-center-estimation errors of up to 101.1 ± 11.2 mm, across joints. Conclusion: These tools offer clinicians and patients the opportunity to estimate movement with research-grade accuracy, without the need for time-consuming calibration steps or the high costs associated with commercial products such as Theia3D or Xsens, helping democratize the diagnosis, prognosis, and treatment of neuromusculoskeletal conditions.

A Review of Deep Learning Techniques for Markerless Human Motion on Synthetic Datasets

Learning a Deep Motion Interpolation Network for Human Skeleton Animations

Marker-Less 3d Human Motion Capture With Monocular Image Sequence And Height-Maps

Anatomical-Marker-Driven 3D Markerless Human Motion Capture

Model-Based Markerless Human Body Motion Capture using Multiple Cameras

A review of 3D human pose estimation algorithms for markerless motion capture

3D Human Mesh Estimation from Virtual Markers

DeepLabCut: markerless pose estimation of user-defined body parts with deep learning

Dynamic Human Body Reconstruction and Motion Tracking with Low-Cost Depth Cameras

A Survey on Deep Learning for Skeleton-Based Human Animation

Multi-view Human Motion Capture with an Improved Deformation Skin Model

Human Motion Estimation from a Reduced Marker Set.

Deep 3D human pose estimation: A review

Towards Accurate Markerless Human Shape and Pose Estimation over Time

Animation of 3D Human Model Using Markerless Motion Capture Applied To Sports

Marker-less Motion Capture Technology Based on Binocular Stereo Vision and Deep Learning

Markerless Body Motion Capturing for 3D Character Animation based on Multi-view Cameras

Markerless Human Body Motion Capture Using Markov Random Field and Dynamic Graph Cuts

Markerless Human Body Motion Capture Using Multiple Cameras

Markerless Motion Tracking With Noisy Video and IMU Data

MODEL-BASEDMARKERLESS HUMAN BODY MOTION CAPTURE USINGMULTIPLE CAMERAS