General-purpose foundation models have become increasingly important in the
field of artificial intelligence. While self-supervised learning (SSL) and
Masked Image Modeling (MIM) have led to promising results in building such
foundation models for remote sensing, these models primarily learn low-level
features, require annotated data for fine-tuning, and not applicable for
retrieval and zero-shot applications due to the lack of language understanding.
In response to these limitations, we propose RemoteCLIP, the first
vision-language foundation model for remote sensing that aims to learn robust
visual features with rich semantics, as well as aligned text embeddings for
seamless downstream application. To address the scarcity of pre-training data,
we leverage data scaling, converting heterogeneous annotations based on
Box-to-Caption (B2C) and Mask-to-Box (M2B) conversion, and further
incorporating UAV imagery, resulting a 12xlarger pretraining dataset.
RemoteCLIP can be applied to a variety of downstream tasks, including zero-shot
image classification, linear probing, k-NN classification, few-shot
classification, image-text retrieval, and object counting. Evaluations on 16
datasets, including a newly introduced RemoteCount benchmark to test the object
counting ability, show that RemoteCLIP consistently outperforms baseline
foundation models across different model scales. Impressively, RemoteCLIP
outperform previous SoTA by 9.14% mean recall on RSICD dataset and by 8.92% on
RSICD dataset. For zero-shot classification, our RemoteCLIP outperform CLIP
baseline by up to 6.39% average accuracy on 12 downstream datasets.

RemoteCLIP 是第一个用于遥感领域的视觉 - 语言基础模型，利用数据扩充和转换方法进行预训练，可用于零样本分类、图像文本检索和物体计数等任务，并在 16 个数据集上均优于基线模型。

RemoteCLIP: 远程感知的视觉语言基础模型

RemoteCLIP: A Vision Language Foundation Model for Remote Sensing

Deep Metric Learning (DML) provides a crucial tool for visual similarity and
zero-shot applications by learning generalizing embedding spaces, although
recent work in DML has shown strong performance saturation across training
objectives. However, generalization capacity is known to scale with the
embedding space dimensionality. Unfortunately, high dimensional embeddings also
create higher retrieval cost for downstream applications. To remedy this, we
propose \emph{Simultaneous Similarity-based Self-distillation (S2SD). S2SD
extends DML with knowledge distillation from auxiliary, high-dimensional
embedding and feature spaces to leverage complementary context during training
while retaining test-time cost and with negligible changes to the training
time. Experiments and ablations across different objectives and standard
benchmarks show S2SD offers notable improvements of up to 7% in Recall@1, while
also setting a new state-of-the-art. Code available at
this https URL

本文提出了一种名为 S2SD 的新方法，将基于相似性的自我蒸馏应用于 Deep Metric Learning 模型中，通过辅助高维度嵌入和特征空间的知识蒸馏，在保持测试时间成本不变的情况下，实现推理性能的显著提升。实验结果表明，该方法能够有效地提高 Recall@1 指标，同时也创造了新的最优性能记录。