NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction

Yiming Wang,Qin Han,Marc Habermann,Kostas Daniilidis,Christian Theobalt,Lingjie Liu

2023-11-17

Abstract:Recent methods for neural surface representation and rendering, for example NeuS, have demonstrated the remarkably high-quality reconstruction of static scenes. However, the training of NeuS takes an extremely long time (8 hours), which makes it almost impossible to apply them to dynamic scenes with thousands of frames. We propose a fast neural surface reconstruction approach, called NeuS2, which achieves two orders of magnitude improvement in terms of acceleration without compromising reconstruction quality. To accelerate the training process, we parameterize a neural surface representation by multi-resolution hash encodings and present a novel lightweight calculation of second-order derivatives tailored to our networks to leverage CUDA parallelism, achieving a factor two speed up. To further stabilize and expedite training, a progressive learning strategy is proposed to optimize multi-resolution hash encodings from coarse to fine. We extend our method for fast training of dynamic scenes, with a proposed incremental training strategy and a novel global transformation prediction component, which allow our method to handle challenging long sequences with large movements and deformations. Our experiments on various datasets demonstrate that NeuS2 significantly outperforms the state-of-the-arts in both surface reconstruction accuracy and training speed for both static and dynamic scenes. The code is available at our website: <a class="link-external link-https" href="https://vcai.mpi-inf.mpg.de/projects/NeuS2/" rel="external noopener nofollow">this https URL</a> .

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The paper aims to address two major issues in neural surface reconstruction: slow training speed and the challenges of dynamic scene reconstruction.

For static scenes, existing methods like NeuS can produce high-quality reconstruction results, but their training process is very time-consuming (approximately 8 hours). Therefore, the paper proposes a new method called NeuS2, which can complete the reconstruction of static objects in a few minutes, significantly improving training speed while maintaining reconstruction quality.

For dynamic scenes, existing technologies struggle to efficiently handle long sequences containing a large number of frames and perform poorly when dealing with significant motion or deformation. NeuS2 addresses these issues by introducing an incremental training strategy and a global transformation prediction component, achieving a fast reconstruction speed of about 20 seconds per frame, making it suitable for long sequence reconstruction with large motion and deformation.

In summary, the main goal of NeuS2 is to significantly improve the speed of neural surface reconstruction for both static and dynamic scenes while ensuring reconstruction quality.