Abstract:The revolutionary capabilities of large language models (LLMs) have paved the way for multimodal large language models (MLLMs) and fostered diverse applications across various specialized domains. In the remote sensing (RS) field, however, the diverse geographical landscapes and varied objects in RS imagery are not adequately considered in recent MLLM endeavors. To bridge this gap, we construct a large-scale RS image-text dataset, LHRS-Align, and an informative RS-specific instruction dataset, LHRS-Instruct, leveraging the extensive volunteered geographic information (VGI) and globally available RS images. Building on this foundation, we introduce LHRS-Bot, an MLLM tailored for RS image understanding through a novel multi-level vision-language alignment strategy and a curriculum learning method. Additionally, we introduce LHRS-Bench, a benchmark for thoroughly evaluating MLLMs' abilities in RS image understanding. Comprehensive experiments demonstrate that LHRS-Bot exhibits a profound understanding of RS images and the ability to perform nuanced reasoning within the RS domain.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in the field of remote sensing (RS) image understanding, the existing multimodal large language models (MLLMs) fail to fully consider the diverse geographical landscapes and different objects in RS images, resulting in poor performance in RS image understanding tasks. Specifically, the paper points out: 1. **Complexity of RS images**: RS images contain complex geographical landscapes and multiple types of objects, and these objects vary greatly at different visual scales, which poses a challenge to overall image understanding. 2. **Limitations of existing datasets**: The existing public RS datasets fail to fully utilize RS features on a global scale, resulting in the inability to comprehensively inject RS visual knowledge into LLMs. 3. **Insufficient visual - language alignment**: Current methods mainly focus on high - level visual semantics and ignore the importance of different levels of visual information for achieving comprehensive alignment between vision and language. To solve these problems, the paper makes the following main contributions: 1. **Constructing a large - scale RS image - text dataset**: The authors construct a large - scale RS image - text dataset named LHRS - Align. By pairing RS images with geographical information in the OpenStreetMap (OSM) database, high - quality image - text pairs are generated. 2. **Creating an RS - specific instruction dataset**: The authors also create a multimodal instruction - following dataset named LHRS - Instruct for RS image understanding tasks, which contains complex visual reasoning data. 3. **Proposing an RS - specific MLLM**: Based on the above datasets, the authors propose an RS - specific MLLM named LHRS - Bot, which adopts a novel multi - level visual - language alignment strategy and curriculum learning method to improve the performance of RS image understanding. 4. **Establishing an evaluation benchmark in the RS field**: The authors construct an evaluation benchmark named LHRS - Bench for comprehensively evaluating MLLMs in the RS field, covering multiple evaluation dimensions and sub - categories. Through these contributions, the paper aims to improve the performance of MLLMs in RS image understanding tasks, especially in detecting complex objects, participating in human conversations, and extracting insights from RS images.

LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model

LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretation

EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain

RS-LLaVA: A Large Vision-Language Model for Joint Captioning and Question Answering in Remote Sensing Imagery

RSGPT: A Remote Sensing Vision Language Model and Benchmark

EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain

RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing

RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Experts

Vision-Language Models in Remote Sensing: Current progress and future trends

Advancements in Visual Language Models for Remote Sensing: Datasets, Capabilities, and Enhancement Techniques

RS-GPT4V: A Unified Multimodal Instruction-Following Dataset for Remote Sensing Image Understanding

SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model

RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models

GeoGround: A Unified Large Vision-Language Model. for Remote Sensing Visual Grounding

Large Language Models for Robotics: Opportunities, Challenges, and Perspectives

RS-Agent: Automating Remote Sensing Tasks through Intelligent Agents

RoboLLM: Robotic Vision Tasks Grounded on Multimodal Large Language Models

LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding

LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models

Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment