Abstract:Robustness is a crucial factor for the successful deployment of robots in unstructured environments, particularly in the domain of Simultaneous Localization and Mapping (SLAM). Simulation-based benchmarks have emerged as a highly scalable approach for robustness evaluation compared to real-world data collection. However, crafting a challenging and controllable noisy world with diverse perturbations remains relatively under-explored. To this end, we propose a novel, customizable pipeline for noisy data synthesis, aimed at assessing the resilience of multi-modal SLAM models against various perturbations. This pipeline incorporates customizable hardware setups, software components, and perturbed environments. In particular, we introduce comprehensive perturbation taxonomy along with a perturbation composition toolbox, allowing the transformation of clean simulations into challenging noisy environments. Utilizing the pipeline, we instantiate the Robust-SLAM benchmark, which includes diverse perturbation types, to evaluate the risk tolerance of existing advanced multi-modal SLAM models. Our extensive analysis uncovers the susceptibilities of existing SLAM models to real-world disturbance, despite their demonstrated accuracy in standard benchmarks. Our perturbation synthesis toolbox, SLAM robustness evaluation pipeline, and Robust-SLAM benchmark will be made publicly available at https://github.com/Xiaohao-Xu/SLAM-under-Perturbation/.

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to address the issue of robustness evaluation of Simultaneous Localization and Mapping (SLAM) systems when deploying robots in complex and dynamic environments. Specifically, the paper proposes a customizable noise data generation pipeline to evaluate the robustness of multimodal SLAM models under various disturbances. ### Background and Motivation 1. **Necessity of Robustness**: - In complex and dynamic environments, the robustness of robots is crucial for their effective operation. - The robustness of SLAM systems in handling sensor failures and environmental changes is key to ensuring long-term performance. 2. **Limitations of Existing Methods**: - Existing SLAM evaluations mainly rely on real-world datasets, but these datasets have limitations in scale and diversity. - While simulated benchmarks offer greater data scalability and diversity, introducing diverse and controllable disturbances in simulated environments remains a challenge. ### Proposed Method 1. **Customizable Noise Data Generation Pipeline**: - This pipeline includes customizable hardware setups, software components, and disturbance environments. - A comprehensive disturbance taxonomy and disturbance synthesis toolbox are introduced, which can transform clean simulated environments into challenging noisy environments. 2. **Robust-SLAM Benchmark**: - Using the above pipeline, a large-scale Robust-SLAM benchmark was generated, containing 1,000 sequences and nearly 2 million RGBD frames. - This benchmark covers various types of disturbances to evaluate the risk tolerance of existing state-of-the-art multimodal SLAM models. ### Main Contributions 1. **Proposed a disturbance taxonomy for SLAM in dynamic and unstructured environments**. 2. **Introduced a noise data generation pipeline, utilizing a disturbance synthesis toolbox for customizable SLAM robustness evaluation**. 3. **Created a large-scale Robust-SLAM dataset, containing various disturbance types, to evaluate SLAM robustness under different sensor failures and dynamic movements**. 4. **Systematically studied and analyzed the robustness of existing SLAM models under isolated disturbance types**. ### Conclusion Through this study, the authors reveal that existing SLAM models perform well in standard benchmarks but exhibit vulnerabilities and tendencies to fail when facing sensor-related or trajectory-related disturbances. This finding provides important guidance for developing more robust SLAM systems.

Customizable Perturbation Synthesis for Robust SLAM Benchmarking

From Perfect to Noisy World Simulation: Customizable Embodied Multi-modal Perturbations for SLAM Robustness Benchmarking

Robust SLAM Systems: Are We There Yet?

Characterizing SLAM Benchmarks and Methods for the Robust Perception Age

Are We Ready for Robust and Resilient SLAM? A Framework For Quantitative Characterization of SLAM Datasets

Robust Monocular SLAM in Dynamic Environments

$\text{R}^2$-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbations

Task-driven SLAM Benchmarking

Improving Model Robustness by Adaptively Correcting Perturbation Levels with Active Queries.

SubT-MRS Dataset: Pushing SLAM Towards All-weather Environments

RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning

PALoc: Robust Prior-assisted Trajectory Generation for Benchmarking

A Framework for Reproducible Benchmarking and Performance Diagnosis of SLAM Systems

Benchmarking the Robustness of LiDAR Semantic Segmentation Models

PALoc: Advancing SLAM Benchmarking with Prior-Assisted 6-DoF Trajectory Generation and Uncertainty Estimation

Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Space

VarBench: Robust Language Model Benchmarking Through Dynamic Variable Perturbation

From Adversarial Arms Race to Model-centric Evaluation: Motivating a Unified Automatic Robustness Evaluation Framework

Evaluating the Robustness of LiDAR-based 3D Obstacles Detection and Its Impacts on Autonomous Driving Systems