Arnab Dey,Cheng-You Lu,Andrew I. Comport,Srinath Sridhar,Chin-Teng Lin,Jean Martinet
Abstract:Recent advancements in radiance field rendering show promising results in 3D scene representation, where Gaussian splatting-based techniques emerge as state-of-the-art due to their quality and efficiency. Gaussian splatting is widely used for various applications, including 3D human representation. However, previous 3D Gaussian splatting methods either use parametric body models as additional information or fail to provide any underlying structure, like human biomechanical features, which are essential for different applications. In this paper, we present a novel approach called HFGaussian that can estimate novel views and human features, such as the 3D skeleton, 3D key points, and dense pose, from sparse input images in real time at 25 FPS. The proposed method leverages generalizable Gaussian splatting technique to represent the human subject and its associated features, enabling efficient and generalizable reconstruction. By incorporating a pose regression network and the feature splatting technique with Gaussian splatting, HFGaussian demonstrates improved capabilities over existing 3D human methods, showcasing the potential of 3D human representations with integrated biomechanics. We thoroughly evaluate our HFGaussian method against the latest state-of-the-art techniques in human Gaussian splatting and pose estimation, demonstrating its real-time, state-of-the-art performance.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to estimate 3D human body representations with integrated biomechanical features from sparse multi - view images under real - time conditions. Specifically, the existing 3D Gaussian Splatting (3DGS) techniques for 3D human body reconstruction either rely on parametric body models or are unable to provide any underlying biomechanical structures, such as 3D skeletons, 3D key points and dense postures, which are crucial for different applications. Therefore, the paper proposes a new method - HFGaussian, aiming to overcome these limitations and achieve efficient and general - purpose reconstruction of 3D human bodies and their related biomechanical features.
### Main Objectives
1. **Real - time 3D human body representation**: HFGaussian can estimate new views and human body features, such as 3D skeletons, 3D key points and dense postures, from sparse input images under real - time conditions (25 FPS).
2. **Integrated biomechanical features**: By combining the pose regression network and the feature splatting technique, HFGaussian can estimate the biomechanical features of the human body more accurately, rather than just the appearance.
3. **Generality**: HFGaussian can be generalized to unseen human body data without fine - tuning, thus improving the applicability and flexibility of the method.
### Solutions
- **Gaussian Splatting Technique**: Use the Gaussian Splatting technique to represent the human body and its related features, making the representation more efficient and general - purpose.
- **Pose Regression Network**: Introduce a new pose regression network, based on DGCNN and PointNet, for estimating 3D poses from partial point cloud data.
- **Feature Splatting**: Through the feature splatting technique, map the Gaussian features to the image plane and use the Alpha - compositing technique to fuse feature vectors, thereby generating a continuous surface embedding.
### Experimental Verification
- **Dataset**: The paper uses a custom - made dataset, which extends the GPS - Gaussian dataset by adding ground - truth information such as key points and dense postures.
- **Performance Evaluation**: By comparing with existing state - of - the - art methods (such as ENeRF, GPS - Gaussian, GHNeRF), it shows the competitiveness and superiority of HFGaussian in new view synthesis, 3D/2D pose estimation, etc.
### Main Contributions
- Propose a new general - purpose method, HFGaussian, which can estimate human body features and 3D poses under real - time conditions.
- Through the pose regression network and the feature splatting technique, achieve efficient estimation of 3D poses and human body features.
- The experimental results on multiple datasets verify the effectiveness and generalization ability of the method.
In conclusion, this paper proposes an innovative method, which solves the limitations of the existing 3D Gaussian Splatting techniques in human body reconstruction and provides a new solution for real - time 3D human body representation and biomechanical feature estimation.