Abstract:This study introduces a novel approach to online embedding of multi-scale CLIP (Contrastive Language-Image Pre-Training) features into 3D maps. By harnessing CLIP, this methodology surpasses the constraints of conventional vocabulary-limited methods and enables the incorporation of semantic information into the resultant maps. While recent approaches have explored the embedding of multi-modal features in maps, they often impose significant computational costs, lacking practicality for exploring unfamiliar environments in real time. Our approach tackles these challenges by efficiently computing and embedding multi-scale CLIP features, thereby facilitating the exploration of unfamiliar environments through real-time map generation. Moreover, the embedding CLIP features into the resultant maps makes offline retrieval via linguistic queries feasible. In essence, our approach simultaneously achieves real-time object search and mapping of unfamiliar environments. Additionally, we propose a zero-shot object-goal navigation system based on our mapping approach, and we validate its efficacy through object-goal navigation, offline object retrieval, and multi-object-goal navigation in both simulated environments and real robot experiments. The findings demonstrate that our method not only exhibits swifter performance than state-of-the-art mapping methods but also surpasses them in terms of the success rate of object-goal navigation tasks.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in complex and unknown environments, how to efficiently embed multi - scale CLIP (Contrastive Language - Image Pretraining) features into 3D maps in real - time, and achieve object retrieval and navigation based on natural language queries. Specifically, the paper mainly focuses on the following aspects: 1. **Vocabulary Limitation**: Traditional methods rely on a fixed vocabulary for object detection and semantic segmentation, and are unable to capture the diverse semantic information in the real - world environment. 2. **Computational Complexity**: Existing methods often face high computational costs when embedding multi - modal features, and it is difficult to achieve real - time applications, especially when exploring unfamiliar environments. 3. **Multi - scale Adaptability**: Single - scale feature embedding methods lack flexibility and cannot adapt to query requirements at different scales. For example, a sink in the kitchen is a "sink" at a small scale, but represents a "kitchen" at a larger scale. To solve these problems, the author proposes a novel method to achieve improvement in the following ways: - **Multi - scale CLIP Feature Embedding**: Use the CLIP model to extract multi - scale visual - language features and efficiently embed them into 3D maps. - **Real - time Performance**: Ensure that the method can operate in a real - time environment and support the adaptability to dynamic environmental changes. - **Zero - shot Navigation**: Develop a zero - shot object - goal navigation system based on the generated map, which can handle open - vocabulary queries. ### Specific Problem Summary 1. **Vocabulary Limitation Problem**: By using the CLIP model, this method can go beyond the traditional fixed vocabulary and support the embedding of semantic information in open vocabularies. 2. **Computational Efficiency Problem**: Through batch processing and multi - scale feature extraction, this method significantly reduces the computational cost and achieves efficient real - time embedding. 3. **Multi - scale Adaptability Problem**: Through multi - scale feature embedding, this method can adapt to query requirements at different scales, improving the practicality and flexibility of the map. These improvements enable this method not only to achieve real - time object search and mapping in unknown environments, but also to enhance the practical application value of the map through the offline retrieval function.

Online Embedding Multi-Scale CLIP Features into 3D Maps

Integrating Visual and Textual Inputs for Searching Large-Scale Map Collections with CLIP

Object-aware Semantic Mapping of Indoor Scenes Using Octomap

Crossview Mapping with Graph-based Geolocalization on City-Scale Street Maps

CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition

Large-Scale 3D Semantic Mapping Using Monocular Vision

Multi-Robot Collaborative Mapping with Integrated Point-Line Features for Visual SLAM

Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery

CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP

Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes

Building and optimization of 3D semantic map based on Lidar and camera fusion

GenMapping: Unleashing the Potential of Inverse Perspective Mapping for Robust Online HD Map Construction

LoCUS: Learning Multiscale 3D-consistent Features from Posed Images

PointCLIP: Point Cloud Understanding by CLIP

CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP

Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Models

An Approach for Construct Semantic Map with Scene Classification and Object Semantic Segmentation

CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data

SeMLaPS: Real-time Semantic Mapping with Latent Prior Networks and Quasi-Planar Segmentation

Clio: Real-time Task-Driven Open-Set 3D Scene Graphs