Progress in Neural Radiance Field and Its Application in Large-Scale Real-Scene 3D Visualization
ZHAO Qiang,SHE Jiangfeng,WAN Qifeng,HE Lixia,LI Sirui,WU Shuangpin
DOI: https://doi.org/10.11834/jrs.20232536
2024-01-01
Abstract:Geographical real-scene 3D scenes are an important national digital infrastructure,which extends geographic information from 2D to 3D.Real-scene 3D data are stored and expressed in the form of an explicit 3D model,which has the problems of large amount of data and rough visualization effect.Neural Radiance Field(NeRF),realizing differentiable rendering based on neural implicit volume representation,is an innovative approach of high-quality view synthesis.First proposed by Mildenhall et al.(2020),NeRF has become one of the hottest research direction in the field of computer vision due to its realistic view synthesis effect.A large amount of literature about NeRF have been published since NeRF was proposed,and the application of NeRF in large-scale real-scene 3D visualization has begun to attract the attention of some published papers. View synthesis,which uses sparse 2D images to generate realistic new views at any viewpoint in 3D space without the reconstruction of 3D models,is a novel way to realize the representation of 3D scenes.The development of view synthesis technology has gone through several stages:image mosaicking,3D model reprojection,view interpolation,and volume representing technology.NeRF,as an innovative approach of view synthesis,samples 5D coordinates(location and viewing direction)along camera rays,feeds those locations into a multilayer perceptron network to produce color and volume density,and uses volume rendering techniques to composite these values into a new image.NeRF not only produces remarkably higher-quality rendering than prior volumetric approaches but also requires just a fraction of the storage cost of other sampled volumetric representations.However,it faces problems such as requirements for high quality of source data,failure to support dynamic objects,low efficiency in processing,and single type of render target.Moreover,NeRF-related research are mostly conducted based on laboratory environment or standardized data at present.Due to these drawbacks,many obstacles need to be overcome before applying NeRF to large-scale real-scene 3D visualization. This paper reviews the workaround of unbounded scene,aliasing,luminosity consistency,scene relighting,and visibility field in the Block-NeRF algorithm,a variant of NeRF that can represent large-scale environments(Tancik et al.2022).The Block-NeRF algorithm splits the environment into a set of Block-NeRFs that can be independently trained in parallel and composited during inference,and it selects relevant Block-NeRFs for rendering,which are then composited smoothly when traversing the scene.To aid with this compositing,Block-NeRF optimizes the appearance codes to match lighting conditions.It trains individual Block-NeRFs using techniques such as appearance embeddings,learned pose refinement,exposure input,transient objects,and visibility prediction. Nowadays,the studies of applying NeRF to large-scale real-scene 3D visualization are being conducted extensively,which has attracted much attention and plays a pioneering,leading role in further research.The results of these studies achieve the most basic 3D visualization large-scale real-scene effect,but they are limited by some conditions and their universality needs to be strongly improved.Because NeRF is still far from practical applications in producing large-scale real-scene 3D visualization,any slight progressive exploration is likely to become a continuous research hotspot.This paper identifies the challenges of NeRF research,including multisource data fusion,visual effect optimization,and virtual environment perception,which need more research.