GPT4GEO: How a Language Model Sees the World's Geography

Jonathan Roberts,Timo Lüddecke,Sowmen Das,Kai Han,Samuel Albanie
DOI: https://doi.org/10.48550/arXiv.2306.00020
2023-05-31
Abstract:Large language models (LLMs) have shown remarkable capabilities across a broad range of tasks involving question answering and the generation of coherent text and code. Comprehensively understanding the strengths and weaknesses of LLMs is beneficial for safety, downstream applications and improving performance. In this work, we investigate the degree to which GPT-4 has acquired factual geographic knowledge and is capable of using this knowledge for interpretative reasoning, which is especially important for applications that involve geographic data, such as geospatial analysis, supply chain management, and disaster response. To this end, we design and conduct a series of diverse experiments, starting from factual tasks such as location, distance and elevation estimation to more complex questions such as generating country outlines and travel networks, route finding under constraints and supply chain analysis. We provide a broad characterisation of what GPT-4 (without plugins or Internet access) knows about the world, highlighting both potentially surprising capabilities but also limitations.
Computation and Language,Artificial Intelligence,Machine Learning
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to evaluate the understanding and application ability of the large - scale language model GPT - 4 in terms of geographical knowledge. Specifically, the authors designed a series of experiments to test GPT - 4's knowledge of world geographical facts and its ability to use this knowledge for explanatory reasoning. This includes from simple geographical location, distance and altitude estimation to more complex tasks, such as generating national outlines, travel networks, path planning and supply chain analysis, etc. Through these experiments, the research aims to comprehensively characterize GPT - 4's geographical knowledge of the world, reveal its potential capabilities and limitations, and thus provide references for applications related to geographical data, such as in fields like geospatial analysis, supply chain management and disaster response.