Abstract:This paper introduces a novel continual learning framework for synthesising novel views of multiple scenes, learning multiple 3D scenes incrementally, and updating the network parameters only with the training data of the upcoming new scene. We build on Neural Radiance Fields (NeRF), which uses multi‐layer perceptron to model the density and radiance field of a scene as the implicit function. While NeRF and its extensions have shown a powerful capability of rendering photo‐realistic novel views in a single 3D scene, managing these growing 3D NeRF assets efficiently is a new scientific problem. Very few works focus on the efficient representation or continuous learning capability of multiple scenes, which is crucial for the practical applications of NeRF. To achieve these goals, our key idea is to represent multiple scenes as the linear combination of a cross‐scene weight matrix and a set of scene‐specific weight matrices generated from a global parameter generator. Furthermore, we propose an uncertain surface knowledge distillation strategy to transfer the radiance field knowledge of previous scenes to the new model. Representing multiple 3D scenes with such weight matrices significantly reduces memory requirements. At the same time, the uncertain surface distillation strategy greatly overcomes the catastrophic forgetting problem and maintains the photo‐realistic rendering quality of previous scenes. Experiments show that the proposed approach achieves state‐of‐the‐art rendering quality of continual learning NeRF on NeRF‐Synthetic, LLFF, and TanksAndTemples datasets while preserving extra low storage cost.

CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting

CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental Learning

$C^{3}$-NeRF: Modeling Multiple Scenes via Conditional-cum-Continual Neural Radiance Fields

Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representations

3D Multi-scene Stylization Based on Conditional Neural Radiance Fields

EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision

Car-Studio: Learning Car Radiance Fields from Single-View and Endless In-the-wild Images

MapNeRF: Incorporating Map Priors into Neural Radiance Fields for Driving View Simulation

NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields

SCARF: Scalable Continual Learning Framework for Memory‐efficient Multiple Neural Radiance Fields

Car-Studio: Learning Car Radiance Fields From Single-View and Unlimited In-the-Wild Images

Lightning NeRF: Efficient Hybrid Scene Representation for Autonomous Driving

Instant Continual Learning of Neural Radiance Fields

LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes

An Exploration of Neural Radiance Field Scene Reconstruction: Synthetic, Real-world and Dynamic Scenes

Towards Embodied Neural Radiance Fields

Ced-NeRF: A Compact and Efficient Method for Dynamic Neural Radiance Fields

Multi-Level Neural Scene Graphs for Dynamic Urban Environments

ACFNeRF: Accelerating and Cache-Free Neural Rendering Via Point Cloud-Based Distance Fields.

DriveEnv-NeRF: Exploration of A NeRF-Based Autonomous Driving Environment for Real-World Performance Validation

PC-NeRF: Parent-Child Neural Radiance Fields Using Sparse LiDAR Frames in Autonomous Driving Environments