Noise-in, Bias-out: Balanced and Real-time MoCap Solving

Georgios Albanis,Nikolaos Zioulis,Spyridon Thermos,Anargyros Chatzitofis,Kostas Kolomvatsos

2023-09-26

Abstract:Real-time optical Motion Capture (MoCap) systems have not benefited from the advances in modern data-driven modeling. In this work we apply machine learning to solve noisy unstructured marker estimates in real-time and deliver robust marker-based MoCap even when using sparse affordable sensors. To achieve this we focus on a number of challenges related to model training, namely the sourcing of training data and their long-tailed distribution. Leveraging representation learning we design a technique for imbalanced regression that requires no additional data or labels and improves the performance of our model in rare and challenging poses. By relying on a unified representation, we show that training such a model is not bound to high-end MoCap training data acquisition, and exploit the advances in marker-less MoCap to acquire the necessary data. Finally, we take a step towards richer and affordable MoCap by adapting a body model-based inverse kinematics solution to account for measurement and inference uncertainty, further improving performance and robustness. Project page: <a class="link-external link-https" href="https://moverseai.github.io/noise-tail" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition,Graphics,Machine Learning

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on several key challenges encountered in real - time optical motion capture (MoCap) systems:

1. **Data Skewness and Long - Tail Distribution**: Human motion data is inherently redundant, and some postures occur far less frequently than others, forming a long - tail distribution of data. This uneven distribution causes machine - learning models to be easily biased towards common postures during the training process, and perform poorly on rare or challenging postures.

2. **Noise Processing**: Even high - end MoCap systems will generate different levels of noise, which may result from information loss (such as marker point swapping, occlusion, ghosting, etc.) or measurement errors (such as jitter, position offset). Existing methods usually assume that the input data is of high quality, but when using consumer - level sensors, the noise level of the input data will be higher, which requires the model to have stronger denoising capabilities.

3. **Training Data Acquisition**: Most existing works rely on high - end MoCap systems to obtain training data. This process is not only costly, time - consuming and labor - intensive, but also difficult to scale up on a large scale. The paper proposes a new method. By using a parameterized human body model and modern marker - less capture techniques, high - quality training data can be obtained without relying on high - end MoCap systems.

4. **Model Robustness and Real - Time Performance**: The paper aims to develop a MoCap solution that can operate in real - time while maintaining high precision and robustness, and can perform well even in the face of high noise levels and sparse sensor inputs.

To address the above challenges, the paper proposes the following technical solutions:

- **Imbalanced Regression Technique**: Solve the problems of data redundancy and long - tail distribution through representation learning, joint over - sampling and utility - based regression.
- **Noise - Aware Body Shape and Pose Solver**: Model the measurement uncertainty area during the optimization process to improve the model's robustness to noise.
- **Real - Time Inference Capability**: Demonstrate an artifact - free MoCap solution model that can run at 60Hz on a system with only 3 consumer - level sensors.
- **Cold - Start Data - Driven Optical MoCap Model Using a Parameterized Human Body Model**: Use the marker - less capture method to obtain data and solve the cold - start problem of data - driven MoCap solution.

In summary, the main objective of this paper is to develop a MoCap framework that can overcome data skewness, handle high - noise inputs, obtain training data without the need for high - end MoCap systems, and operate efficiently in a real - time environment.