Abstract:Cross-modality registration between 2D images from cameras and 3D point clouds from LiDARs is a crucial task in computer vision and robotic. Previous methods estimate 2D-3D correspondences by matching point and pixel patterns learned by neural networks, and use Perspective-n-Points (PnP) to estimate rigid transformation during post-processing. However, these methods struggle to map points and pixels to a shared latent space robustly since points and pixels have very different characteristics with patterns learned in different manners (MLP and CNN), and they also fail to construct supervision directly on the transformation since the PnP is non-differentiable, which leads to unstable registration results. To address these problems, we propose to learn a structured cross-modality latent space to represent pixel features and 3D features via a differentiable probabilistic PnP solver. Specifically, we design a triplet network to learn VoxelPoint-to-Pixel matching, where we represent 3D elements using both voxels and points to learn the cross-modality latent space with pixels. We design both the voxel and pixel branch based on CNNs to operate convolutions on voxels/pixels represented in grids, and integrate an additional point branch to regain the information lost during voxelization. We train our framework end-to-end by imposing supervisions directly on the predicted pose distribution with a probabilistic PnP solver. To explore distinctive patterns of cross-modality features, we design a novel loss with adaptive-weighted optimization for cross-modality feature description. The experimental results on KITTI and nuScenes datasets show significant improvements over the state-of-the-art methods. The code and models are available at <a class="link-external link-https" href="https://github.com/junshengzhou/VP2P-Match" rel="external noopener nofollow">this https URL</a>.

CMRNext: Camera to LiDAR Matching in the Wild for Localization and Extrinsic Calibration

Leveraging Local Planar Motion Property for Robust Visual Matching and Localization.

2-Entity Random Sample Consensus for Robust Visual Localization: Framework, Methods, and Verifications

Automatic Target-Less Camera-LiDAR Calibration From Motion and Deep Point Correspondences

Monocular Camera Localization in Prior LiDAR Maps with 2D-3D Line Correspondences

Monocular Visual Place Recognition in LiDAR Maps via Cross-Modal State Space Model and Multi-View Matching

Online,Target-Free LiDAR-Camera Extrinsic Calibration via Cross-Modal Mask Matching

Cross-Modal Monocular Localization in Prior LiDAR Maps Utilizing Semantic Consistency.

SemCal: Semantic LiDAR-Camera Calibration using Neural MutualInformation Estimator

Attention-Enhanced Cross-modal Localization Between Spherical Images and Point Clouds

MonoNext: A 3D Monocular Object Detection with ConvNext

Validation & Exploration of Multimodal Deep-Learning Camera-Lidar Calibration models

Real-time LIDAR localization in natural and urban environments

Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching

CP+: Camera Poses Augmentation with Large-scale LiDAR Maps

LHMap-loc: Cross-Modal Monocular Localization Using LiDAR Point Cloud Heat Map

Deep Robust Multi-Robot Re-localisation in Natural Environments

Self-supervised Learning of LiDAR 3D Point Clouds via 2D-3D Neural Calibration

C2L-PR: Cross-modal Camera-to-LiDAR Place Recognition Via Modality Alignment and Orientation Voting

Learning a Novel LiDAR Submap-Based Observation Model for Global Positioning in Long-Term Changing Environments

End-to-end Automatic Targetless Calibration Between Low-resolution LiDARs and Cameras Based on Cross-modal Attention