Enhancing Worldwide Image Geolocation by Ensembling Satellite-Based Ground-Level Attribute Predictors

Michael J. Bianco,David Eigen,Michael Gormish

2024-09-18

Abstract:We examine the challenge of estimating the location of a single ground-level image in the absence of GPS or other location metadata. Currently, geolocation systems are evaluated by measuring the Great Circle Distance between the predicted location and ground truth. Because this measurement only uses a single point, it cannot assess the distribution of predictions by geolocation systems. Evaluation of a distribution of potential locations (areas) is required when there are follow-on procedures to further narrow down or verify the location. This is especially important in poorly-sampled regions e.g. rural and wilderness areas. In this paper, we introduce a novel metric, Recall vs Area (RvA), which measures the accuracy of estimated distributions of locations. RvA treats image geolocation results similarly to document retrieval, measuring recall as a function of area: For a ranked list of (possibly discontiguous) predicted regions, we measure the area required for accumulated regions to contain the ground truth coordinate. This produces a curve similar to a precision-recall curve, where "precision" is replaced by square kilometers area, enabling evaluation for different downstream search area budgets. Following from this view of the problem, we then examine an ensembling approach to global-scale image geolocation, which incorporates information from multiple sources, and can readily incorporate multiple models, attribute predictors, and data sources. We study its effectiveness by combining the geolocation models GeoEstimation and the current state-of-the-art, GeoCLIP, with attribute predictors based on Oak Ridge National Laboratory LandScan and European Space Agency Climate Change Initiative Land Cover. We find significant improvements in image geolocation for areas that are under-represented in the training set, particularly non-urban areas, on both Im2GPS3k and Street View images.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper mainly addresses the challenge of estimating the geographical location of a single ground - level image in the absence of GPS or other location metadata. Specifically, the authors focus on the following key issues:

1. **Limitations of existing evaluation methods**:
- Current geolocation systems usually evaluate performance by measuring the great circle distance (GCD) between the predicted location and the true location. This method uses only a single point for evaluation and cannot measure the quality of the predicted area set or heat map.
- In rural areas, wilderness, and under - sampled areas, it can be very difficult to find the exact location, so a method that can evaluate larger areas is needed.

2. **Improving evaluation metrics**:
- To better evaluate geolocation performance in these situations, the authors introduce a new evaluation metric - recall vs area (RvA). RvA measures the accuracy of the predicted distribution. It treats the geolocation result as a document retrieval problem and calculates the recall by accumulating the area of the predicted region.
- The specific formula is:
\[
\text{recall}(\alpha)=\frac{1}{|X|}\sum_{i}1[y_{i}\in\{a_{ik}:\sum_{k' = 1}^{k}a_{ik'}\leq\alpha\}]
\]
where $\alpha$ is the area threshold, $y_{i}$ is the true location of the $i$-th image, and $a_{ik}$ is the location area predicted by the model.

3. **Multi - source information fusion**:
- To improve the accuracy of global - scale image geolocation, especially in non - urban areas that are under - represented in the training set, the authors propose a simple integration method. This method combines information from multiple sources, including multiple models, attribute predictors, and data sources.
- For example, combining GeoEstimation and the current state - of - the - art GeoCLIP model, as well as attribute predictors based on Oak Ridge National Laboratory LandScan and European Space Agency Climate Change Initiative Land Cover.

4. **Meeting the challenges of under - sampled areas**:
- Geotagged images in the training data set are mainly concentrated in cities and other densely populated areas, resulting in insufficient samples in many rural and remote areas. To this end, the authors use the global coverage information provided by satellite data products (such as LandScan and Land Cover) to expand to those areas with fewer ground - level image samples.

Through the above methods, the paper aims to improve the robustness and generalization ability of the geolocation system, especially its performance in dealing with under - sampled areas.