Foundation models have excelled in various tasks but are often evaluated on general benchmarks. The adaptation of these models for specific domains, such as remote sensing imagery, remains an underexplored area. In remote sensing, precise building instance segmentation is vital for applications like urban planning. While Convolutional Neural Networks (CNNs) perform well, their generalization can be limited. For this aim, we present a novel approach to adapt foundation models to address existing models' generalization dropback. Among several models, our focus centers on the Segment Anything Model (SAM), a potent foundation model renowned for its prowess in class-agnostic image segmentation capabilities. We start by identifying the limitations of SAM, revealing its suboptimal performance when applied to remote sensing imagery. Moreover, SAM does not offer recognition abilities and thus fails to classify and tag localized objects. To address these limitations, we introduce different prompting strategies, including integrating a pre-trained CNN as a prompt generator. This novel approach augments SAM with recognition abilities, a first of its kind. We evaluated our method on three remote sensing datasets, including the WHU Buildings dataset, the Massachusetts Buildings dataset, and the AICrowd Mapping Challenge. For out-of-distribution performance on the WHU dataset, we achieve a 5.47% increase in IoU and a 4.81% improvement in F1-score. For in-distribution performance on the WHU dataset, we observe a 2.72% and 1.58% increase in True-Positive-IoU and True-Positive-F1 score, respectively. We intend to release our code repository, hoping to inspire further exploration of foundation models for domain-specific tasks within the remote sensing community.

我们提出了一种新颖的方法来针对现有模型的泛化性能下降问题，将基础模型自适应于特定领域，如遥感图像。我们通过集成预训练的卷积神经网络作为提示生成器，为Segment Anything Model增加了识别能力，从而改善了其在遥感图像上的表现。通过在三个遥感数据集上评估我们的方法，包括WHU Buildings数据集、Massachusetts Buildings数据集和AICrowd Mapping Challenge数据集，我们观察到我们的方法在不同分布的性能上取得了显著提高，并计划发布我们的代码库，以促进遥感领域对基础模型进行更多领域特定任务的探索。

使用SAM进行建筑物分割模型的零样本精细调整