Abstract:Extracting robust and discriminative local features from images plays a vital role for long term visual localization, whose challenges are mainly caused by the severe appearance differences between matching images due to the day-night illuminations, seasonal changes, and human activities. Existing solutions resort to jointly learning both keypoints and their descriptors in an end-to-end manner, leveraged on large number of annotations of point correspondence which are harvested from the structure from motion and depth estimation algorithms. While these methods show improved performance over non-deep methods or those two-stage deep methods, i.e., detection and then description, they are still struggled to conquer the problems encountered in long term visual localization. Since the intrinsic semantics are invariant to the local appearance changes, this paper proposes to learn semantic-aware local features in order to improve robustness of local feature matching for long term localization. Based on a state of the art CNN architecture for local feature learning, i.e., ASLFeat, this paper leverages on the semantic information from an off-the-shelf semantic segmentation network to learn semantic-aware feature maps. The learned correspondence-aware feature descriptors and semantic features are then merged to form the final feature descriptors, for which the improved feature matching ability has been observed in experiments. In addition, the learned semantics embedded in the features can be further used to filter out noisy keypoints, leading to additional accuracy improvement and faster matching speed. Experiments on two popular long term visual localization benchmarks (Aachen Day and Night v1.1, Robotcar Seasons) and one challenging indoor benchmark (InLoc) demonstrate encouraging improvements of the localization accuracy over its counterpart and other competitive methods.

Learning Semantic Correspondences from Noisy Data-text Pairs by Local-to-Global Alignments

Joint CRF and Locality-Consistent Dictionary Learning for Semantic Segmentation.

A Local Online Learning Approach for Non-linear Data.

Learning Semantic Alignment Using Global Features and Multi-scale Confidence

Learning Latent Semantic Annotations for Grounding Natural Language to Structured Data

Global and Local Semantic Completion Learning for Vision-Language Pre-training

Learning to Generate Semantic Layouts for Higher Text-Image Correspondence in Text-to-Image Synthesis

Global Structure and Local Semantics-Preserved Embeddings for Entity Alignment

Learning Semantic Correspondence Exploiting an Object-Level Prior

Text-Video Retrieval with Global-Local Semantic Consistent Learning

A Local Information Perception Enhancement–Based Method for Chinese NER

Joint Learning of Semantic Alignment and Object Landmark Detection

Global-Local Interplay in Semantic Alignment for Few-Shot Learning

Multi-Grained Cross-modal Alignment for Learning Open-vocabulary Semantic Segmentation from Text Supervision

Cross-Modal Semantic Alignment before Fusion for Two-Pass End-to-End Spoken Language Understanding

Match me if you can: Semi-Supervised Semantic Correspondence Learning with Unpaired Images

Multi-level multilingual semantic alignment for zero-shot cross-lingual transfer learning

Weakly Supervised Learning of Semantic Correspondence Through Cascaded Online Correspondence Refinement

Learning Global-Local Correspondence with Semantic Bottleneck for Logical Anomaly Detection

A semi-supervised neighborhood matching model for global entity alignment

Learning Semantic-Aware Local Features for Long Term Visual Localization