Abstract:In numerous robotic and autonomous driving tasks, traditional visual SLAM algorithms estimate the camera's position in a scene through sparse feature points and express the map by estimating the depth of sparse point clouds. However, practical applications require SLAM to create dense maps in real time, overcoming the sparsity and occlusion issues of point clouds. Furthermore, it is advantageous for SLAM map to possess an auto-completion capability, where the map can automatically infer and complete the remaining 20% when the camera observes only 80% of an object. Therefore, a more dense and intelligent map representation is needed. In this paper, we propose a Visual–Inertial SLAM with Neural Radiance Fields reconstruction to address the aforementioned challenges. We integrate the traditional rule-based optimization with NeRF. This approach allows for the real-time update of NeRF local functions by rapidly estimating camera motion and sparse feature point depths to reconstruct 3D scenes. To achieve better camera poses and globally consistent map, we address the issue of IMU noise spikes resulting from rapid motion changes, along with handling pose adjustments due to loop closure fusion. Specifically, we employ a form of widening the static noise covariance to refit the dynamic noise covariance. During loop closure fusion, we treat the pose adjustment between pre- and post-loop closure as a spatiotemporal transformation, migrating NeRF parameters from pre- to post- to expedite loop closure adjustments in NeRF mapping. Moreover, we extend this method to scenarios with only grayscale images. By expanding the color channels of grayscale images and conducting linear spatial mapping, we can rapidly reconstruct 3D scenes with only grayscale images. We demonstrate the precision and speed advantages of our method in both RGB and grayscale scenes.

FVLoc-NeRF : Fast Vision-Only Localization Within Neural Radiation Field

NeRF-Loc: Visual Localization with Conditional Neural Radiance Field.

Fast Global Localization on Neural Radiance Field

PNeRFLoc: Visual Localization with Point-based Neural Radiance Fields

Loc-NeRF: Monte Carlo Localization using Neural Radiance Fields

NeRF-Loc: Transformer-Based Object Localization Within Neural Radiance Fields

VRS-NeRF: Visual Relocalization with Sparse Neural Radiance Field

NeRF-VINS: A Real-time Neural Radiance Field Map-based Visual-Inertial Navigation System

NuRF: Nudging the Particle Filter in Radiance Fields for Robot Visual Localization

The NeRFect Match: Exploring NeRF Features for Visual Localization

Matching Query Image Against Selected NeRF Feature for Efficient and Scalable Localization

VI-NeRF-SLAM: a real-time visual–inertial SLAM with NeRF mapping

NeRFVS: Neural Radiance Fields for Free View Synthesis Via Geometry Scaffolds

IOVS4NeRF:Incremental Optimal View Selection for Large-Scale NeRFs

NeRF-VO: Real-Time Sparse Visual Odometry with Neural Radiance Fields

F$^{2}$-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories

NeRF-IBVS: Visual Servo Based on NeRF for Visual Localization and Navigation.

LATITUDE: Robotic Global Localization with Truncated Dynamic Low-pass Filter in City-scale NeRF

CloudLoc-NeRF: Point-cloud Assisted Volume Location for Neural Radiance Fields

NVINS: Robust Visual Inertial Navigation Fused with NeRF-augmented Camera Pose Regressor and Uncertainty Quantification