Abstract:Internet image collections containing photos captured by crowds of photographers show promise for enabling digital exploration of large-scale tourist landmarks. However, prior works focus primarily on geometric reconstruction and visualization, neglecting the key role of language in providing a semantic interface for navigation and fine-grained understanding. In constrained 3D domains, recent methods have leveraged vision-and-language models as a strong prior of 2D visual semantics. While these models display an excellent understanding of broad visual semantics, they struggle with unconstrained photo collections depicting such tourist landmarks, as they lack expert knowledge of the architectural domain. In this work, we present a localization system that connects neural representations of scenes depicting large-scale landmarks with text describing a semantic region within the scene, by harnessing the power of SOTA vision-and-language models with adaptations for understanding landmark scene semantics. To bolster such models with fine-grained knowledge, we leverage large-scale Internet data containing images of similar landmarks along with weakly-related textual information. Our approach is built upon the premise that images physically grounded in space can provide a powerful supervision signal for localizing new concepts, whose semantics may be unlocked from Internet textual metadata with large language models. We use correspondences between views of scenes to bootstrap spatial understanding of these semantics, providing guidance for 3D-compatible segmentation that ultimately lifts to a volumetric scene representation. Our results show that HaLo-NeRF can accurately localize a variety of semantic concepts related to architectural landmarks, surpassing the results of other 3D models as well as strong 2D segmentation baselines. Our project page is at <a class="link-external link-https" href="https://tau-vailab.github.io/HaLo-NeRF/" rel="external noopener nofollow">this https URL</a>.

Learning Spatial Prior With Automatically Labeled Landmarks

Spatial Likelihood Voting with Self-Knowledge Distillation for Weakly Supervised Object Detection.

SLV: Spatial Likelihood Voting for Weakly Supervised Object Detection

LocNet: Global Localization in 3D Point Clouds for Mobile Robots.

Hierarchical Saliency Mapping for Weakly Supervised Object Localization Based on Class Activation Mapping

Automatic Location and Semantic Labeling of Landmarks on 3D Human Body Models

Unsupervised Discovery of Object Landmarks as Structural Representations

Self-Supervised Object Localization with Joint Graph Partition

Learning a unified label space

Adversarial Learning of Structure-Aware Fully Convolutional Networks for Landmark Localization

Combining Deep Learning and Model-Based Methods for Robust Real-Time Semantic Landmark Detection

CurriculumLoc: Enhancing Cross-Domain Geolocalization Through Multistage Refinement

Self-supervised Secondary Landmark Detection via 3D Representation Learning

CurriculumLoc: Enhancing Cross-Domain Geolocalization through Multi-Stage Refinement

Learning Local Semantic Region Activations for Weakly Supervised Object Localization

Salient Region Detection Via Low-Level Features and High-Level Priors

LoCUS: Learning Multiscale 3D-consistent Features from Posed Images

One-Shot Medical Landmark Localization by Edge-Guided Transform and Noisy Landmark Refinement

Improved Scene Landmark Detection for Camera Localization

Accurate Spatial Positioning of Target Based on the Fusion of Uncalibrated Image and GNSS

HaLo-NeRF: Learning Geometry-Guided Semantics for Exploring Unconstrained Photo Collections