Abstract:Objective: Marker-based motion capture, considered the gold standard in human motion analysis, is expensive and requires trained personnel. Advances in inertial sensing and computer vision offer new opportunities to obtain research-grade assessments in clinics and natural environments. A challenge that discourages clinical adoption, however, is the need for careful sensor-to-body alignment, which slows the data collection process in clinics and is prone to errors when patients take the sensors home. Methods: We propose deep learning models to estimate human movement with noisy data from videos (VideoNet), inertial sensors (IMUNet), and a combination of the two (FusionNet), obviating the need for careful calibration. The video and inertial sensing data used to train the models were generated synthetically from a marker-based motion capture dataset of a broad range of activities and augmented to account for sensor-misplacement and camera-occlusion errors. The models were tested using real data that included walking, jogging, squatting, sit-to-stand, and other activities. Results: On calibrated data, IMUNet was as accurate as state-of-the-art models, while VideoNet and FusionNet reduced mean ± std root-mean-squared errors by 7.6 ± 5.4 ° and 5.9 ± 3.3 °, respectively. Importantly, all the newly proposed models were less sensitive to noise than existing approaches, reducing errors by up to 14.0 ± 5.3 ° for sensor-misplacement errors of up to 30.0 ± 13.7 ° and by up to 7.4 ± 5.5 ° for joint-center-estimation errors of up to 101.1 ± 11.2 mm, across joints. Conclusion: These tools offer clinicians and patients the opportunity to estimate movement with research-grade accuracy, without the need for time-consuming calibration steps or the high costs associated with commercial products such as Theia3D or Xsens, helping democratize the diagnosis, prognosis, and treatment of neuromusculoskeletal conditions.

Human Motion Estimation from a Reduced Marker Set.

Predicting Missing Markers In Human Motion Capture Using L1-Sparse Representation

Motion Imitation of a Humanoid Robot Via Pose Estimation

Marker-Less 3d Human Motion Capture With Monocular Image Sequence And Height-Maps

A 3D Human Motion Refinement Method Based on Sparse Motion Bases Selection.

Multi-view Human Motion Capture with an Improved Deformation Skin Model

Markerless Shape and Motion Capture From Multiview Video Sequences

Towards Accurate Markerless Human Shape and Pose Estimation over Time

Markerless motion capture of multiple characters using multiview image segmentation

MarkerNet: A divide‐and‐conquer solution to motion capture solving from raw markers

Markerless motion capture of interacting characters using multi-view image segmentation

Cross Refinement Techniques for Markerless Human Motion Capture

Capture Dense: Markerless Motion Capture Meets Dense Pose Estimation

Marker Data Enhancement For Markerless Motion Capture

3D Human Mesh Estimation from Virtual Markers

Dynamic Human Body Reconstruction and Motion Tracking with Low-Cost Depth Cameras

Anatomical-Marker-Driven 3D Markerless Human Motion Capture

A Review of Deep Learning Techniques for Markerless Human Motion on Synthetic Datasets

Markerless Motion Tracking With Noisy Video and IMU Data

Marker-Less Human Body Tracking Using Locally Affine Invariant Contour Matching

New multi-view human motion capture framework