Abstract:Light-weight time-of-flight (ToF) depth sensors are compact and cost-efficient, and thus widely used on mobile devices for tasks such as autofocus and obstacle detection. However, due to the sparse and noisy depth measurements, these sensors have rarely been considered for dense geometry reconstruction. In this work, we present the first dense SLAM system with a monocular camera and a light-weight ToF sensor. Specifically, we propose a multi-modal implicit scene representation that supports rendering both the signals from the RGB camera and light-weight ToF sensor which drives the optimization by comparing with the raw sensor inputs. Moreover, in order to guarantee successful pose tracking and reconstruction, we exploit a predicted depth as an intermediate supervision and develop a coarse-to-fine optimization strategy for efficient learning of the implicit representation. At last, the temporal information is explicitly exploited to deal with the noisy signals from light-weight ToF sensors to improve the accuracy and robustness of the system. Experiments demonstrate that our system well exploits the signals of light-weight ToF sensors and achieves competitive results both on camera tracking and dense scene reconstruction. Project page: \url{<a class="link-external link-https" href="https://zju3dv.github.io/tof_slam/" rel="external noopener nofollow">this https URL</a>}.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenges encountered in performing Dense Simultaneous Localization and Mapping (Dense SLAM) using lightweight Time - of - Flight (ToF) sensors and monocular cameras. Specifically, due to their compact design, lightweight ToF sensors can only provide low - resolution depth distribution measurements, which makes existing RGB - D dense SLAM systems unable to directly utilize these signals. These problems include: 1. **Low - resolution depth signal**: The depth signals provided by lightweight ToF sensors are very sparse and noisy, and it is difficult to directly use them for high - precision scene reconstruction and camera pose estimation. 2. **Limitations of existing systems**: Existing RGB - D SLAM systems usually require high - resolution depth images as input, and lightweight ToF sensors cannot provide such data. 3. **Multi - modal signal fusion**: How to effectively fuse the RGB images of the monocular camera and the depth signals of the lightweight ToF sensors to achieve accurate camera tracking and dense scene reconstruction. To solve these problems, the paper proposes a new multi - modal implicit scene representation method, which can process RGB images and signals from lightweight ToF sensors simultaneously, and is achieved through the following technical means: - **Multi - modal implicit scene representation**: A multi - modal implicit scene representation is designed to support the rendering of RGB images and signals from lightweight ToF sensors. - **Predicted depth as intermediate supervision**: A depth prediction model is used to generate high - resolution depth maps as intermediate supervision to improve the robustness and accuracy of the system. - **Coarse - to - fine optimization strategy**: A coarse - to - fine optimization strategy is adopted. First, the region - level ToF signals are used for rough optimization, and then pixel - level RGB/depth supervision is added to restore geometric details. - **Temporal filtering technique**: A temporal filtering technique is developed. By fusing historical observations and current observations, the depth prediction module is enhanced, especially when the quality of the original ToF signals is poor. Through these technical means, the paper successfully achieves the goal of performing dense SLAM using only a monocular camera and a lightweight ToF sensor, and demonstrates its competitiveness in camera tracking and scene reconstruction in experiments.

Multi-Modal Neural Radiance Field for Monocular Dense SLAM with a Light-Weight ToF Sensor

MFF-Net: Towards Efficient Monocular Depth Completion With Multi-Modal Feature Fusion

DELTAR: Depth Estimation from a Light-Weight ToF Sensor and RGB Image

NeSLAM: Neural Implicit Mapping and Self-Supervised Feature Tracking With Depth Completion and Denoising

NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields

Robust Monocular SLAM in Dynamic Environments

TöRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis

TivNe-SLAM: Dynamic Mapping and Tracking via Time-Varying Neural Radiance Fields

Real-Time Dense Visual SLAM with Neural Factor Representation

MoD-SLAM: Monocular Dense Mapping for Unbounded 3D Scene Reconstruction

A Monocular Visual SLAM System Augmented by Lightweight Deep Local Feature Extractor Using In-House and Low-Cost LIDAR-camera Integrated Device

VI-NeRF-SLAM: a real-time visual–inertial SLAM with NeRF mapping

M-DIVO: Multiple ToF RGB-D Cameras Enhanced Depth-Inertial-Visual Odometry

Neural Implicit Representation for Highly Dynamic LiDAR Mapping and Odometry

HI-SLAM: Monocular Real-time Dense Mapping with Hybrid Implicit Fields

A Robust Deep Learning Enhanced Monocular SLAM System for Dynamic Environments

Uni-SLAM: Uncertainty-Aware Neural Implicit SLAM for Real-Time Dense Indoor Scene Reconstruction

SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videos

Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM

Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping for Monocular, Stereo, and RGB-D Cameras

SLAIM: Robust Dense Neural SLAM for Online Tracking and Mapping