Abstract:LiDAR-Camera alignment (LCA) is an important preprocessing procedure for fusing LiDAR and camera data. For it, one key issue is to extract unified cross-modality representation for characterizing the heterogeneous LiDAR and camera data effectively and robustly. The main challenge is to resist the modality gap and visual data degradation during feature learning, while still maintaining strong representative power. To address this, a novel modality adapted local-to-global representation learning method is proposed. The research efforts are paid in 2 main folders via modality adaptation and capturing global spatial context. First for modality gap resistance, LiDAR and camera data is projected into the same depth map domain for unified representation learning. Particularly, LiDAR data is converted to depth map according to pre-acquired extrinsic parameters. Thanks to the recent advantage of deep learning based monocular depth estimation, camera data is transformed into depth map in data driven manner, which is jointly optimized with LCA. Secondly to capture global spatial context, ViT (vision transformer) is introduced to LCA. The concept of LCA token is proposed for aggregating the local spatial patterns to form global spatial representation with transformer encoding. And, it is shared by all the samples. In this way, it can involve global sample-level information to leverage generalization ability. The experiments on KITTI dataset verify superiority of our proposition. Furthermore, the proposed approach is more robust to camera data degeneration (e.g., imaging blurring and noise) often faced by the practical applications. Under some challenging test cases, the performance advancement of our method is over $1.9~cm$ /4.1° on translation / rotation error. While our model size (8.77M) is much smaller than existing methods (e.g., LCCNet of 66.75M). The source code will be released at https://github.com/Zaf233/RLCA upon acceptance.

Attention-Enhanced Cross-modal Localization Between Spherical Images and Point Clouds

Persistent Stereo Visual Localization on Cross-Modal Invariant Map

3D LiDAR-Based Global Localization Using Siamese Neural Network

Multimodal Localization: Stereo over LiDAR Map

LiDAR-Based Global Localization Using Histogram of Orientations of Principal Normals

2-Entity RANSAC for Robust Visual Localization in Changing Environment

LocNet: Global Localization in 3D Point Clouds for Mobile Vehicles

Cross-Modal 2D-3D Localization with Single-Modal Query

2D-3D Cross-Modality Network for End-to-End Localization with Probabilistic Supervision

Cross-Modal Monocular Localization in Prior LiDAR Maps Utilizing Semantic Consistency.

Monocular Visual Place Recognition in LiDAR Maps via Cross-Modal State Space Model and Multi-View Matching

BEVLOC: End-to-End 6-Dof Localization Via Cross-Modality Correlation under Bird’s Eye View

C2L-PR: Cross-modal Camera-to-LiDAR Place Recognition Via Modality Alignment and Orientation Voting

LHMap-loc: Cross-Modal Monocular Localization Using LiDAR Point Cloud Heat Map

CrossLoc: Scalable Aerial Localization Assisted by Multimodal Synthetic Data

(LC)$^2$: LiDAR-Camera Loop Constraints For Cross-Modal Place Recognition

CMLocate: A Cross‐modal Automatic Visual Geo‐localization Framework for a Natural Environment Without GNSS Information

A Hybrid Approach for Cross-modality Pose Estimation Between Image and Point Cloud

Robust LiDAR-Camera Alignment with Modality Adapted Local-to-Global Representation

Cross-Modal Visual Relocalization in Prior LiDAR Maps Utilizing Intensity Textures

Monocular Camera Localization in Prior LiDAR Maps with 2D-3D Line Correspondences