Foundation models have excelled in various tasks but are often evaluated on
general benchmarks. The adaptation of these models for specific domains, such
as remote sensing imagery, remains an underexplored area. In remote sensing,
precise building instance segmentation is vital for applications like urban
planning. While Convolutional Neural Networks (CNNs) perform well, their
generalization can be limited. For this aim, we present a novel approach to
adapt foundation models to address existing models' generalization dropback.
Among several models, our focus centers on the Segment Anything Model (SAM), a
potent foundation model renowned for its prowess in class-agnostic image
segmentation capabilities. We start by identifying the limitations of SAM,
revealing its suboptimal performance when applied to remote sensing imagery.
Moreover, SAM does not offer recognition abilities and thus fails to classify
and tag localized objects. To address these limitations, we introduce different
prompting strategies, including integrating a pre-trained CNN as a prompt
generator. This novel approach augments SAM with recognition abilities, a first
of its kind. We evaluated our method on three remote sensing datasets,
including the WHU Buildings dataset, the Massachusetts Buildings dataset, and
the AICrowd Mapping Challenge. For out-of-distribution performance on the WHU
dataset, we achieve a 5.47% increase in IoU and a 4.81% improvement in
F1-score. For in-distribution performance on the WHU dataset, we observe a
2.72% and 1.58% increase in True-Positive-IoU and True-Positive-F1 score,
respectively. We intend to release our code repository, hoping to inspire
further exploration of foundation models for domain-specific tasks within the
remote sensing community.

我们提出了一种新颖的方法来针对现有模型的泛化性能下降问题，将基础模型自适应于特定领域，如遥感图像。我们通过集成预训练的卷积神经网络作为提示生成器，为 Segment Anything Model 增加了识别能力，从而改善了其在遥感图像上的表现。通过在三个遥感数据集上评估我们的方法，包括 WHU Buildings 数据集、Massachusetts Buildings 数据集和 AICrowd Mapping Challenge 数据集，我们观察到我们的方法在不同分布的性能上取得了显著提高，并计划发布我们的代码库，以促进遥感领域对基础模型进行更多领域特定任务的探索。