Large language models (LLMs) have recently been extended to the
vision-language realm, obtaining impressive general multi-modal capabilities.
However, the exploration of multi-modal large language models (MLLMs) for
remote sensing (RS) data is still in its infancy, and the performance is not
satisfactory. In this work, we introduce SkyEyeGPT, a unified multi-modal large
language model specifically designed for RS vision-language understanding. To
this end, we meticulously curate an RS multi-modal instruction tuning dataset,
including single-task and multi-task conversation instructions. After manual
verification, we obtain a high-quality RS instruction-following dataset with
968k samples. Our research demonstrates that with a simple yet effective
design, SkyEyeGPT works surprisingly well on considerably different tasks
without the need for extra encoding modules. Specifically, after projecting RS
visual features to the language domain via an alignment layer, they are fed
jointly with task-specific instructions into an LLM-based RS decoder to predict
answers for RS open-ended tasks. In addition, we design a two-stage tuning
method to enhance instruction-following and multi-turn dialogue ability at
different granularities. Experiments on 8 datasets for RS vision-language tasks
demonstrate SkyEyeGPT's superiority in image-level and region-level tasks, such
as captioning and visual grounding. In particular, SkyEyeGPT exhibits
encouraging results compared to GPT-4V in some qualitative tests. The online
demo, code, and dataset will be released in
this https URL

SkyEyeGPT 是一个针对远程感知视觉语言理解而设计的统一的多模态大型语言模型，通过对视觉特征与语言域的对齐以及使用 LLM 的 RS 解码器来预测远程感知开放式任务的答案，在 RS 视觉语言任务中表现出卓越的图像级和区域级任务结果。