The remote sensing image intelligence understanding model is undergoing a new
profound paradigm shift which has been promoted by multi-modal large language
model (MLLM), i.e. from the paradigm learning a domain model (LaDM) shifts to
paradigm learning a pre-trained general foundation model followed by an
adaptive domain model (LaGD). Under the new LaGD paradigm, the old datasets,
which have led to advances in RSI intelligence understanding in the last
decade, are no longer suitable for fire-new tasks. We argued that a new dataset
must be designed to lighten tasks with the following features: 1)
Generalization: training model to learn shared knowledge among tasks and to
adapt to different tasks; 2) Understanding complex scenes: training model to
understand the fine-grained attribute of the objects of interest, and to be
able to describe the scene with natural language; 3) Reasoning: training model
to be able to realize high-level visual reasoning. In this paper, we designed a
high-quality, diversified, and unified multimodal instruction-following dataset
for RSI understanding produced by GPT-4V and existing datasets, which we called
RS-GPT4V. To achieve generalization, we used a (Question, Answer) which was
deduced from GPT-4V via instruction-following to unify the tasks such as
captioning and localization; To achieve complex scene, we proposed a
hierarchical instruction description with local strategy in which the
fine-grained attributes of the objects and their spatial relationships are
described and global strategy in which all the local information are integrated
to yield detailed instruction descript; To achieve reasoning, we designed
multiple-turn QA pair to provide the reasoning ability for a model. The
empirical results show that the fine-tuned MLLMs by RS-GPT4V can describe
fine-grained information. The dataset is available at:
this https URL

远程传感图像智能理解模型正在经历一个深刻的新范式转变，即从学习域模型的范式转变为先学习预训练的通用基础模型，然后再进行自适应域模型的范式。在新的自适应域模型范式下，过去十年中已取得远程传感图像智能理解进展的旧数据集不再适用于全新任务。我们认为必须设计一个具备以下特征的新数据集来轻化任务：1) 泛化性：训练模型学习多个任务之间的共享知识，并适应不同的任务；2) 理解复杂场景：训练模型理解感兴趣对象的细粒度属性，并能用自然语言描述场景；3) 推理能力：训练模型能够实现高层次的视觉推理。本文设计了一个由 GPT-4V 和现有数据集共同创建的高质量、多样化、统一的多模式指令跟踪数据集，我们称之为 RS-GPT4V。为了实现泛化性，我们使用了由 GPT-4V 通过指令跟踪引导推导出来的（问题，答案）对来统一诸如字幕和定位等任务；为了实现复杂场景，我们提出了一种具有局部策略的分层指令描述方式，描述了对象的细粒度属性和它们的空间关系，并具有全局策略将所有局部信息集成以生成详细的指令描述；为了实现推理能力，我们设计了多轮问答对来为模型提供推理能力。实证结果表明，通过 RS-GPT4V 微调的多模态大语言模型能够描述细粒度信息。数据集可在此网址获取：https://example.com