Abstract:Cross-modality registration between 2D images from cameras and 3D point clouds from LiDARs is a crucial task in computer vision and robotic. Previous methods estimate 2D-3D correspondences by matching point and pixel patterns learned by neural networks, and use Perspective-n-Points (PnP) to estimate rigid transformation during post-processing. However, these methods struggle to map points and pixels to a shared latent space robustly since points and pixels have very different characteristics with patterns learned in different manners (MLP and CNN), and they also fail to construct supervision directly on the transformation since the PnP is non-differentiable, which leads to unstable registration results. To address these problems, we propose to learn a structured cross-modality latent space to represent pixel features and 3D features via a differentiable probabilistic PnP solver. Specifically, we design a triplet network to learn VoxelPoint-to-Pixel matching, where we represent 3D elements using both voxels and points to learn the cross-modality latent space with pixels. We design both the voxel and pixel branch based on CNNs to operate convolutions on voxels/pixels represented in grids, and integrate an additional point branch to regain the information lost during voxelization. We train our framework end-to-end by imposing supervisions directly on the predicted pose distribution with a probabilistic PnP solver. To explore distinctive patterns of cross-modality features, we design a novel loss with adaptive-weighted optimization for cross-modality feature description. The experimental results on KITTI and nuScenes datasets show significant improvements over the state-of-the-art methods. The code and models are available at <a class="link-external link-https" href="https://github.com/junshengzhou/VP2P-Match" rel="external noopener nofollow">this https URL</a>.

Metric Learning for Patch-Based 3-D Image Registration.

Metric Learning for Image Registration

Metric Learning Via Penalized Optimization.

A Metric Learning Method for Image-based 3D Shape Retrieval

PATS: Patch Area Transportation with Subdivision for Local Feature Matching.

An Automatic Registration Approach to Laser Point Sets Based on Multidiscriminant Parameter Extraction

Metric networks for enhanced perception of non-local semantic information

Multimodal Medical Image Registration Via Common Representations Learning and Differentiable Geometric Constraints

Regularization in Metric Learning for Person Re-Identification

Distance Metric Learning from Uncertain Side Information with Application to Automated Photo Tagging

A Deep Metric for Multimodal Registration

Semantic Interaction Meta-Learning Based on Patch Matching Metric

Image Patch-Matching with Graph-Based Learning in Street Scenes

3D Focusing-and-Matching Network for Multi-Instance Point Cloud Registration

Bidirectional Patch-based Correlations with Local Rigidity for Global Nonrigid Registration

Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching

Vision-Based Pose Optimization Using Learned Metrics

Metric Learning Based Multi-Patch Ensemble for High Precision Face Verification

3D Meta-Registration: Learning to Learn Registration of 3D Point Clouds

EMD Metric Learning

2D3D-MATR: 2D-3D Matching Transformer for Detection-free Registration between Images and Point Clouds