Abstract:Camera relocalization is a challenging task to estimate camera pose within a known scene, with wide applications in the fields of Virtual Reality (VR), Augmented Reality (AR), robotics, and etc. Most existing learning-based methods invariably utilize all the information within an image for pose estimation. Although these methods have demonstrated leading pose accuracy in some cases, they are still far from being sufficient to handle the robustness under challenging viewpoints with less impacts on the localization accuracy for viewpoints that are easier to localize. In this paper, we propose a novel two-branch camera pose estimation framework: one branch utilizes keypoint-guided partial scene coordinate regression, while the other employs full scene coordinate regression to assess the credibility of image poses, thereby enabling more accurate camera localization. In particular, we devise a keypoint selection method predicated on matching rates which is designed to measure the matching quality between a 3D keypoint and 2D keypoints across views. With these selected 3D keypoints, we can generate 2D supervision mask with the ground-truth camera pose to supervise the keypoint prediction from the keypoint selection network. Meanwhile, we further refine the 2D supervision mask through the optimization with reprojection errors on the scene coordinate network, which estimates the scene coordinates for points within the scene that truly warrant attention, also enhances the localization performance. We also introduce a gated camera pose estimation strategy on the two-branch pose estimation framework, employing an updated keypoint selection network for images with higher credibility and a more robust network for difficult viewpoints. By adopting an effective curriculum learning scheme, we achieve higher accuracy within a training span of just 20 minutes. Our method's superior performance is validated through rigorous experimentation. The code is released at https://github.com/DUT-ICCD/KP-Guided-Reloc.

Keypoint Matching via Random Network Consensus

2-Entity Random Sample Consensus for Robust Visual Localization: Framework, Methods, and Verifications

Dynamic Keypoint Detection Network for Image Matching

A Concurrent Multiscale Detector for End-to-End Image Matching

Repeatable Adaptive Keypoint Detection Via Self-Supervised Learning

Learning Feature Matching via Matchable Keypoint-Assisted Graph Neural Network

DDM-NET: End-to-end learning of keypoint feature Detection, Description and Matching for 3D localization

Weakly Supervised Learning for Image Keypoint Matching Using Graph Convolutional Networks.

A Region-Based Descriptor Network for Uniformly Sampled Keypoints

Reinforced Feature Points: Optimizing Feature Detection and Description for a High-Level Task

P2-Net - Joint Description and Detection of Local Features for Pixel and Point Matching.

Exploring Matching Rates: from Keypoint Selection to Camera Relocalization

From Independence to Cooperation: A Novel Keypoint Extraction Framework for Autonomous Systems

Multi-Level Feature Aggregation-Based Joint Keypoint Detection and Description

LifelongGlue: Keypoint Matching for 3D Reconstruction with Continual Neural Networks

Realtime CNN-based Keypoint Detector with Sobel Filter and CNN-based Descriptor Trained with Keypoint Candidates

2D3D-MatchNet: Learning to Match Keypoints Across 2D Image and 3D Point Cloud

SEKD: Self-Evolving Keypoint Detection and Description

Learning 2D-3D Correspondences To Solve The Blind Perspective-n-Point Problem

VIEWPOINT ESTIMATION IN IMAGES BY A KEY-POINT BASED DEEP NEURAL NETWORK

Real-time CNN-based Keypoint Detector with Sobel Filter and Descriptor Trained with Keypoint Candidates