Recently, remote sensing image captioning has gained significant attention in
the remote sensing community. Due to the significant differences in spatial
resolution of remote sensing images, existing methods in this field have
predominantly concentrated on the fine-grained extraction of remote sensing
image features, but they cannot effectively handle the semantic consistency
between visual features and textual features. To efficiently align the
image-text, we propose a novel two-stage vision-language pre-training-based
approach to bootstrap interactive image-text alignment for remote sensing image
captioning, called BITA, which relies on the design of a lightweight
interactive Fourier Transformer to better align remote sensing image-text
features. The Fourier layer in the interactive Fourier Transformer is capable
of extracting multi-scale features of remote sensing images in the frequency
domain, thereby reducing the redundancy of remote sensing visual features.
Specifically, the first stage involves preliminary alignment through image-text
contrastive learning, which aligns the learned multi-scale remote sensing
features from the interactive Fourier Transformer with textual features. In the
second stage, the interactive Fourier Transformer connects the frozen image
encoder with a large language model. Then, prefix causal language modeling is
utilized to guide the text generation process using visual features.
Ultimately, across the UCM-caption, RSICD, and NWPU-caption datasets, the
experimental results clearly demonstrate that BITA outperforms other advanced
comparative approaches. The code is available at
this https URL

基于视觉语言预训练的新型交互式傅里叶变换方法，用于实现远程感知图像字幕生成，通过对遥感图像和文本特征的更好对齐，提高了语义一致性，并在多个数据集上表现出超过其他方法的优越性能。

遥感图像字幕生成中的交互式图像 - 文本对齐引导

Bootstrapping Interactive Image-Text Alignment for Remote Sensing Image  Captioning

The emergence of large-scale large language models, with GPT-4 as a prominent
example, has significantly propelled the rapid advancement of artificial
general intelligence and sparked the revolution of Artificial Intelligence 2.0.
In the realm of remote sensing (RS), there is a growing interest in developing
large vision language models (VLMs) specifically tailored for data analysis in
this domain. However, current research predominantly revolves around visual
recognition tasks, lacking comprehensive, large-scale image-text datasets that
are aligned and suitable for training large VLMs, which poses significant
challenges to effectively training such models for RS applications. In computer
vision, recent research has demonstrated that fine-tuning large vision language
models on small-scale, high-quality datasets can yield impressive performance
in visual and language understanding. These results are comparable to
state-of-the-art VLMs trained from scratch on massive amounts of data, such as
GPT-4. Inspired by this captivating idea, in this work, we build a high-quality
Remote Sensing Image Captioning dataset (RSICap) that facilitates the
development of large VLMs in the RS field. Unlike previous RS datasets that
either employ model-generated captions or short descriptions, RSICap comprises
2,585 human-annotated captions with rich and high-quality information. This
dataset offers detailed descriptions for each image, encompassing scene
descriptions (e.g., residential area, airport, or farmland) as well as object
information (e.g., color, shape, quantity, absolute position, etc). To
facilitate the evaluation of VLMs in the field of RS, we also provide a
benchmark evaluation dataset called RSIEval. This dataset consists of
human-annotated captions and visual question-answer pairs, allowing for a
comprehensive assessment of VLMs in the context of RS.

大规模语言模型的出现显著推动了人工智能的快速发展，并引发了人工智能 2.0 的革命。远程感知领域对于开发专门针对数据分析的大规模视觉语言模型越来越感兴趣。然而，目前的研究主要集中在视觉识别任务上，缺乏对齐并适用于训练大规模视觉语言模型的全面、大规模图像 - 文本数据集，这对于有效训练此类模型构成了重大挑战。本研究构建了一个高质量的远程感知图像字幕数据集（RSICap），用于促进 RS 领域大规模视觉语言模型的发展。与以往通过模型生成的字幕或简短描述的数据集不同，RSICap 包括 2,585 个人工注释的字幕，具备丰富和高质量的信息。该数据集为每个图像提供了详细的描述，包括场景描述（例如居住区、机场或农田）以及对象信息（例如颜色、形状、数量、绝对位置等）。为了促进在 RS 领域中对视觉语言模型的评估，我们还提供了一个基准评估数据集 RSIEval，它包括人工注释的字幕和视觉问答对，可以全面评估在 RS 背景下的视觉语言模型。