Towards Embodied Neural Radiance Fields

Xinghang Li,Di Guo,Huaping Liu

DOI: https://doi.org/10.1109/cyber63482.2024.10748705

2024-01-01

Abstract:Neural Radiance Fields (NeRF) is a powerful rendering approach that represents scenes implicitly. By utilizing 5D coordinates along camera rays, NeRF models volume density and radiance field, yielding impressive rendering results. However, NeRF heavily relies on a large number of images from various viewpoints for neural network training, and its image collection and model training stages are separate. To address this, we introduce an embodied NeRF framework that leverages the agent's embodiment to automatically collect suitable images during navigation, facilitating effective NeRF model training. Our framework incorporates a novel navigation policy that guides the agent to explore the scene region by region, generated using a scene graph aware region division method. The captured images are then used to train NeRF models for the scene. To generalize the learned navigation policy to new scenes, we employ a student-forcing learning method with carefullydesigned demonstration paths. Extensive experiments are conducted in both simulated and physical environments to validate the effectiveness of our embodied NeRF framework.