3D Human Scan With A Moving Event Camera

Kai Kohyama,Shintaro Shiba,Yoshimitsu Aoki

2024-04-16

Abstract:Capturing a 3D human body is one of the important tasks in computer vision with a wide range of applications such as virtual reality and sports analysis. However, conventional frame cameras are limited by their temporal resolution and dynamic range, which imposes constraints in real-world application setups. Event cameras have the advantages of high temporal resolution and high dynamic range (HDR), but the development of event-based methods is necessary to handle data with different characteristics. This paper proposes a novel event-based method for 3D pose estimation and human mesh recovery. Prior work on event-based human mesh recovery require frames (images) as well as event data. The proposed method solely relies on events; it carves 3D voxels by moving the event camera around a stationary body, reconstructs the human pose and mesh by attenuated rays, and fit statistical body models, preserving high-frequency details. The experimental results show that the proposed method outperforms conventional frame-based methods in the estimation accuracy of both pose and body mesh. We also demonstrate results in challenging situations where a conventional camera has motion blur. This is the first to demonstrate event-only human mesh recovery, and we hope that it is the first step toward achieving robust and accurate 3D human body scanning from vision sensors.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper primarily aims to address the following issues:

1. **Limitations of Traditional Frame Cameras**: Traditional frame cameras face the problem of insufficient temporal resolution when dealing with fast movements (such as sports activities), leading to image blur. Additionally, in low-light environments, the shutter speed needs to be adjusted, which limits the dynamic range.

2. **Advantages and Challenges of Event Cameras**: Event cameras have high temporal resolution and high dynamic range (HDR), making them very suitable for capturing rapidly changing scenes. However, the data they generate is different from traditional frame cameras, requiring the development of specialized methods to process this data.

3. **3D Human Scanning and Reconstruction**: To address the above challenges, this paper proposes a new method based on event cameras for 3D human pose estimation and human mesh recovery. This method relies solely on event data, scanning a stationary human by moving the event camera around them to carve out 3D voxels. It uses attenuation ray techniques to enhance detail retention and ultimately fits a statistical human model to obtain accurate human pose and mesh information.

4. **Improvements Over Existing Work**: Compared to previous work that required combining event data and frame images, this method achieves the goal of recovering the human mesh solely from event data for the first time. Furthermore, experimental results show that this method outperforms traditional frame-based methods in terms of estimation accuracy and performs well even in the presence of motion blur.

In summary, this paper aims to overcome the limitations of traditional frame cameras through an innovative approach, leveraging the advantages of event cameras to achieve more accurate and robust 3D human scanning.