Abstract:Image-to-point cloud (I2P) registration is a fundamental task for robots and autonomous vehicles to achieve cross-modality data fusion and localization. Current I2P registration methods primarily focus on estimating correspondences at the point or pixel level, often neglecting global alignment. As a result, I2P matching can easily converge to a local optimum if it lacks high-level guidance from global constraints. To improve the success rate and general robustness, this paper introduces CoFiI2P, a novel I2P registration network that extracts correspondences in a coarse-to-fine manner. First, the image and point cloud data are processed through a two-stream encoder-decoder network for hierarchical feature extraction. Second, a coarse-to-fine matching module is designed to leverage these features and establish robust feature correspondences. Specifically, In the coarse matching phase, a novel I2P transformer module is employed to capture both homogeneous and heterogeneous global information from the image and point cloud data. This enables the estimation of coarse super-point/super-pixel matching pairs with discriminative descriptors. In the fine matching module, point/pixel pairs are established with the guidance of super-point/super-pixel correspondences. Finally, based on matching pairs, the transform matrix is estimated with the EPnP-RANSAC algorithm. Experiments conducted on the KITTI Odometry dataset demonstrate that CoFiI2P achieves impressive results, with a relative rotation error (RRE) of 1.14 degrees and a relative translation error (RTE) of 0.29 meters, while maintaining real-time speed.Additional experiments on the Nuscenes datasets confirm our method's generalizability. The project page is available at \url{<a class="link-external link-https" href="https://whu-usi3dv.github.io/CoFiI2P" rel="external noopener nofollow">this https URL</a>}.

What problem does this paper attempt to address?

The paper attempts to address the challenges in achieving image-to-point cloud (I2P) registration in robots and autonomous vehicles. Specifically: 1. **Limitations of existing methods**: Current I2P registration methods mainly focus on point or pixel-level correspondence estimation, often neglecting global alignment, which leads to matches easily converging to local optima. 2. **Need for cross-modal data fusion**: To achieve cross-modal data fusion and localization, a method is needed that can establish accurate pose transformation from the image coordinate system to the pre-built point cloud coordinate system. To address these issues, the paper proposes the CoFiI2P network, a novel coarse-to-fine I2P registration method that incrementally improves registration accuracy through a staged approach, avoiding local optima and significantly enhancing the success rate and robustness of registration. Experimental results on the KITTI Odometry and Nuscenes datasets show that CoFiI2P achieves significant improvements in both relative rotation error (RRE) and relative translation error (RTE), with improvements of 84% and 89% respectively compared to the current best methods.

CoFiI2P: Coarse-to-Fine Correspondences for Image-to-Point Cloud Registration