Abstract:The aim of this work is to introduce MaRF, a novel framework able to synthesize the Martian environment using several collections of images from rover cameras. The idea is to generate a 3D scene of Mars' surface to address key challenges in planetary surface exploration such as: planetary geology, simulated navigation and shape analysis. Although there exist different methods to enable a 3D reconstruction of Mars' surface, they rely on classical computer graphics techniques that incur high amounts of computational resources during the reconstruction process, and have limitations with generalizing reconstructions to unseen scenes and adapting to new images coming from rover cameras. The proposed framework solves the aforementioned limitations by exploiting Neural Radiance Fields (NeRFs), a method that synthesize complex scenes by optimizing a continuous volumetric scene function using a sparse set of images. To speed up the learning process, we replaced the sparse set of rover images with their neural graphics primitives (NGPs), a set of vectors of fixed length that are learned to preserve the information of the original images in a significantly smaller size. In the experimental section, we demonstrate the environments created from actual Mars datasets captured by Curiosity rover, Perseverance rover and Ingenuity helicopter, all of which are available on the Planetary Data System (PDS).

What problem does this paper attempt to address?

This paper aims to introduce MaRF (Mars Radiance Fields), a novel framework capable of synthesizing 3D scenes of the Martian environment using multiple sets of images captured by rover cameras. Specifically, this framework aims to address key challenges in planetary surface exploration, such as planetary geology, simulated navigation, and shape analysis. Although there are currently different methods to achieve 3D reconstruction of the Martian surface, these methods rely on traditional computer graphics techniques, which require substantial computational resources during the reconstruction process and have limitations in generalizing to unseen scenes and adapting to new images. MaRF addresses these issues by utilizing Neural Radiance Fields (NeRF). NeRF is a method for synthesizing complex scenes by optimizing a continuous volumetric scene function, trained using a sparse set of images. To accelerate the learning process, the authors replaced the sparse rover images with Neural Graphics Primitives (NGP). NGP is a set of fixed-length vectors that, once learned, can retain the information of the original images at a significantly smaller size. In the experimental section, the authors demonstrate environments created from actual Martian datasets captured by the Curiosity rover, Perseverance rover, and Ingenuity helicopter. Overall, MaRF, by combining mixed reality technology and neural networks, not only efficiently generates 3D views of the Martian surface but also provides higher scientific returns in areas such as scientific exploration and robotic navigation. Additionally, this framework can be used to compress rover images, encoding the entire scene into a Multi-Layer Perceptron (MLP), with a storage size several orders of magnitude smaller than a single PDS image while retaining the same amount of information as the original image set.

MaRF: Representing Mars as Neural Radiance Fields

NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

BRDF-NeRF: Neural Radiance Fields with Optical Satellite Images and BRDF Modelling

Baking Neural Radiance Fields for Real-Time View Synthesis

NeRFMeshing: Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshes

Neural Radiance Fields (NeRFs): A Review and Some Recent Developments

Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representations

Neural Surface Reconstruction and Rendering for LiDAR-Visual Systems

MMNeRF: Multi-Modal and Multi-View Optimized Cross-Scene Neural Radiance Fields

NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild

Vision-based Neural Scene Representations for Spacecraft

Mapping "Brain Terrain" Regions on Mars using Deep Learning

Urban Radiance Fields

NeRF-MS: Neural Radiance Fields with Multi-Sequence.

Di-NeRF: Distributed NeRF for Collaborative Learning with Relative Pose Refinement

CtrlNeRF: The Generative Neural Radiation Fields for the Controllable Synthesis of High-fidelity 3D-Aware Images

NeRFuser: Large-Scale Scene Representation by NeRF Fusion

NEURAL RADIANCE FIELDS (NERF) FOR MULTI-SCALE 3D MODELING OF CULTURAL HERITAGE ARTIFACTS

SPARF: Neural Radiance Fields from Sparse and Noisy Poses

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields

Multi-tiling Neural Radiance Field (NeRF) -- Geometric Assessment on Large-scale Aerial Datasets