Comparing metrics for evaluating 3D map quality in natural environments

Stéphanie Aravecchia,Marianne Clausel,Cédric Pradalier
DOI: https://doi.org/10.1016/j.robot.2023.104617
IF: 3.7
2024-01-06
Robotics and Autonomous Systems
Abstract:In this study, we focus on addressing the challenge of measuring the 3D-map quality in natural environments. Specifically, we consider scenarios where the map is built using a robot's 3D-Lidar point cloud observations, with potential uncertainty in the robot localization. When considering a natural environment, such as a park or a forest, unstructured by nature, another difficulty arises: the data becomes extremely sparse. As a result, measuring the map quality becomes even more challenging. This study aims to compare the effectiveness of various metrics in measuring the 3D-map quality. Firstly, we evaluate these metrics in a controlled experimental setup, where the reconstructed map is created by progressively degrading the reference map using different degradation models. Secondly, we compare their ability to measure 3D-map quality at a local level, across various simulated environments, ranging from structured to unstructured. Finally, we conduct a qualitative comparison to demonstrate the robustness of certain metrics to noise in the robot localization. This qualitative comparison is done both in simulation and in a real world experiment. Ultimately, we synthesize the properties of these metrics and provide practical recommendations for their selection.
robotics,automation & control systems,computer science, artificial intelligence
What problem does this paper attempt to address?