FusionRF: High-Fidelity Satellite Neural Radiance Fields from Multispectral and Panchromatic Acquisitions

Michael Sprintson,Rama Chellappa,Cheng Peng

2024-09-23

Abstract:We introduce FusionRF, a novel neural rendering terrain reconstruction method from optically unprocessed satellite imagery. While previous methods depend on external pansharpening methods to fuse low resolution multispectral imagery and high resolution panchromatic imagery, FusionRF directly performs reconstruction based on optically unprocessed acquisitions with no prior knowledge. This is accomplished through the addition of a sharpening kernel which models the resolution loss in multispectral images. Additionally, novel modal embeddings allow the model to perform image fusion as a bottleneck to novel view synthesis. We evaluate our method on multispectral and panchromatic satellite images from the WorldView-3 satellite in various locations, and FusionRF outperforms previous State-of-The-Art methods in depth reconstruction on unprocessed imagery, renders sharp training and novel views, and retains multi-spectral information.

Computer Vision and Pattern Recognition,Image and Video Processing

What problem does this paper attempt to address?

The paper aims to address the issue of resolution loss in terrain reconstruction from satellite images and proposes a novel neural rendering method called FusionRF. Specifically, traditional satellite image processing methods rely on external preprocessing steps (such as pansharpening, which fuses multispectral and panchromatic images), and these steps may lead to spectral artifacts and other errors. FusionRF directly reconstructs from raw multispectral and panchromatic images without optical preprocessing, modeling the spatial resolution loss of multispectral images by introducing a sharpening kernel and achieving image fusion using a novel modality embedding technique. This method not only avoids the problems brought by traditional pansharpening methods but also generates high-fidelity novel view synthesis images without the need for additional preprocessing steps, improving reconstruction accuracy while preserving multispectral information. Furthermore, FusionRF demonstrates superior performance in depth reconstruction, image clarity, and pansharpening compared to existing methods in multiple experiments.