NerfBridge: Bringing Real-time, Online Neural Radiance Field Training to Robotics

Javier Yu,Jun En Low,Keiko Nagami,Mac Schwager

2023-05-17

Abstract:This work was presented at the IEEE International Conference on Robotics and Automation 2023 Workshop on Unconventional Spatial Representations. Neural radiance fields (NeRFs) are a class of implicit scene representations that model 3D environments from color images. NeRFs are expressive, and can model the complex and multi-scale geometry of real world environments, which potentially makes them a powerful tool for robotics applications. Modern NeRF training libraries can generate a photo-realistic NeRF from a static data set in just a few seconds, but are designed for offline use and require a slow pose optimization pre-computation step. In this work we propose NerfBridge, an open-source bridge between the Robot Operating System (ROS) and the popular Nerfstudio library for real-time, online training of NeRFs from a stream of images. NerfBridge enables rapid development of research on applications of NeRFs in robotics by providing an extensible interface to the efficient training pipelines and model libraries provided by Nerfstudio. As an example use case we outline a hardware setup that can be used NerfBridge to train a NeRF from images captured by a camera mounted to a quadrotor in both indoor and outdoor environments. For accompanying video <a class="link-external link-https" href="https://youtu.be/EH0SLn-RcDg" rel="external noopener nofollow">this https URL</a> and code <a class="link-external link-https" href="https://github.com/javieryu/nerf_bridge" rel="external noopener nofollow">this https URL</a>.

Robotics

What problem does this paper attempt to address?

The main goal of this paper is to address the issue of real-time online training of Neural Radiance Fields (NeRF) in robotics. Specifically:

1. **Existing Problem**: Traditional NeRF training libraries, while capable of quickly generating realistic scene representations, are primarily designed for offline applications. This means that the dataset needs to be fully collected before training, and a slow pose optimization preprocessing step is required.

2. **Solution**: The authors propose NerfBridge, an open-source software package for real-time, online NeRF training. It enables real-time training from image streams by connecting the Robot Operating System (ROS) with the popular Nerfstudio library.

3. **Application Scenario**: The paper uses a quadrotor drone as an example to demonstrate how NerfBridge can be used to build NeRF maps in real-time in both indoor and outdoor environments. This real-time training method allows operators to adjust the flight path based on the current quality of the NeRF during flight, thus avoiding the traditional offline workflow that requires multiple landings, image uploads, and NeRF retraining.

4. **Future Directions**: The authors also discuss potential applications of NerfBridge in future research, including combining NeRF with navigation algorithms for robot trajectory optimization and mapping, as well as exploring new information-based keyframe schemes to avoid catastrophic forgetting during the training process.