Abstract:The remote sensing image intelligence understanding model is undergoing a new profound paradigm shift which has been promoted by multi-modal large language model (MLLM), i.e. from the paradigm learning a domain model (LaDM) shifts to paradigm learning a pre-trained general foundation model followed by an adaptive domain model (LaGD). Under the new LaGD paradigm, the old datasets, which have led to advances in RSI intelligence understanding in the last decade, are no longer suitable for fire-new tasks. We argued that a new dataset must be designed to lighten tasks with the following features: 1) Generalization: training model to learn shared knowledge among tasks and to adapt to different tasks; 2) Understanding complex scenes: training model to understand the fine-grained attribute of the objects of interest, and to be able to describe the scene with natural language; 3) Reasoning: training model to be able to realize high-level visual reasoning. In this paper, we designed a high-quality, diversified, and unified multimodal instruction-following dataset for RSI understanding produced by GPT-4V and existing datasets, which we called RS-GPT4V. To achieve generalization, we used a (Question, Answer) which was deduced from GPT-4V via instruction-following to unify the tasks such as captioning and localization; To achieve complex scene, we proposed a hierarchical instruction description with local strategy in which the fine-grained attributes of the objects and their spatial relationships are described and global strategy in which all the local information are integrated to yield detailed instruction descript; To achieve reasoning, we designed multiple-turn QA pair to provide the reasoning ability for a model. The empirical results show that the fine-tuned MLLMs by RS-GPT4V can describe fine-grained information. The dataset is available at: <a class="link-external link-https" href="https://github.com/GeoX-Lab/RS-GPT4V" rel="external noopener nofollow">this https URL</a>.

Panoptic Perception: A Novel Task and Fine-grained Dataset for Universal Remote Sensing Image Interpretation

Panoptic Perception: A Novel Task and Fine-Grained Dataset for Universal Remote Sensing Image Interpretation

Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images

Aerial-PASS: Panoramic Annular Scene Segmentation in Drone Videos

DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization

Towards Panoptic 3D Parsing for Single Image in the Wild

PanDepth: Joint Panoptic Segmentation and Depth Completion

360+x: A Panoptic Multi-modal Scene Understanding Dataset

RS-GPT4V: A Unified Multimodal Instruction-Following Dataset for Remote Sensing Image Understanding

See, Perceive, and Answer: A Unified Benchmark for High-Resolution Postdisaster Evaluation in Remote Sensing Images

SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding

VEnvision3D: A Synthetic Perception Dataset for 3D Multi-Task Model Research

PanopticNeRF-360: Panoramic 3D-to-2D Label Transfer in Urban Scenes

ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation

PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentation

Panoramic Panoptic Segmentation: Insights Into Surrounding Parsing for Mobile Agents via Unsupervised Contrastive Learning

RSPS-SAM: A Remote Sensing Image Panoptic Segmentation Method Based on SAM

Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning

RSGPT: A Remote Sensing Vision Language Model and Benchmark

Towards Robust Pansharpening: A Large-Scale High-Resolution Multi-Scene Dataset and Novel Approach

Large-scale Video Panoptic Segmentation in the Wild: A Benchmark