Abstract:Cross-modality registration between 2D images from cameras and 3D point clouds from LiDARs is a crucial task in computer vision and robotic. Previous methods estimate 2D-3D correspondences by matching point and pixel patterns learned by neural networks, and use Perspective-n-Points (PnP) to estimate rigid transformation during post-processing. However, these methods struggle to map points and pixels to a shared latent space robustly since points and pixels have very different characteristics with patterns learned in different manners (MLP and CNN), and they also fail to construct supervision directly on the transformation since the PnP is non-differentiable, which leads to unstable registration results. To address these problems, we propose to learn a structured cross-modality latent space to represent pixel features and 3D features via a differentiable probabilistic PnP solver. Specifically, we design a triplet network to learn VoxelPoint-to-Pixel matching, where we represent 3D elements using both voxels and points to learn the cross-modality latent space with pixels. We design both the voxel and pixel branch based on CNNs to operate convolutions on voxels/pixels represented in grids, and integrate an additional point branch to regain the information lost during voxelization. We train our framework end-to-end by imposing supervisions directly on the predicted pose distribution with a probabilistic PnP solver. To explore distinctive patterns of cross-modality features, we design a novel loss with adaptive-weighted optimization for cross-modality feature description. The experimental results on KITTI and nuScenes datasets show significant improvements over the state-of-the-art methods. The code and models are available at <a class="link-external link-https" href="https://github.com/junshengzhou/VP2P-Match" rel="external noopener nofollow">this https URL</a>.

VMB module for Low Overlap Point Cloud Registration

A New Method for Registration of 3d Point Sets with Low Overlapping Ratios

A Unified BEV Model for Joint Learning of 3D Local Features and Overlap Estimation.

Point cloud registration and localization based on voxel plane features

Global Registration of Point Cloud Maps with Low-overlap Regions

Hierarchical and Validated Branch-and-Bound Method for Global Point Cloud Registration

Low Overlapping Point Cloud Registration using Mutual Prior based Completion Network

Overlap Bias Matching is Necessary for Point Cloud Registration

L-PR: Exploiting LiDAR Fiducial Marker for Unordered Low Overlap Multiview Point Cloud Registration

DMS: Low-overlap Registration of 3D Point Clouds with Double-layer Multi-scale Star-graph

Deep learning-based low overlap point cloud registration for complex scenario: The review

Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching

An Efficient and Stable Registration Framework for Large Point Clouds at Two Different Moments

Robust point cloud registration using Hough voting-based correspondence outlier rejection

Efficient low-overlapping point cloud registration based on two-stage network learning

VRHCF: Cross-Source Point Cloud Registration via Voxel Representation and Hierarchical Correspondence Filtering

A Novel Point Cloud Coarse Registration Method Combining 2D image information

Low-Overlap Point Cloud Registration With Transformer

Incremental Multiview Point Cloud Registration

OKR-Net: Overlapping Keypoints Registration Network for Large-Scale LiDAR Point Clouds

Hierarchical Optimization of 3D Point Cloud Registration