We study the potential of noisy labels y to pretrain semantic segmentation models in a multi-modal learning framework for geospatial applications. Specifically, we propose a novel Cross-modal Sample Selection method (CromSS) that utilizes the class distributions P^{(d)}(x,c) over pixels x and classes c modelled by multiple sensors/modalities d of a given geospatial scene. Consistency of predictions across sensors $d$ is jointly informed by the entropy of P^{(d)}(x,c). Noisy label sampling we determine by the confidence of each sensor d in the noisy class label, P^{(d)}(x,c=y(x)). To verify the performance of our approach, we conduct experiments with Sentinel-1 (radar) and Sentinel-2 (optical) satellite imagery from the globally-sampled SSL4EO-S12 dataset. We pair those scenes with 9-class noisy labels sourced from the Google Dynamic World project for pretraining. Transfer learning evaluations (downstream task) on the DFC2020 dataset confirm the effectiveness of the proposed method for remote sensing image segmentation.

我们研究了在多模态学习框架中，利用噪声标签y对语义分割模型进行预训练的潜力，针对地理空间应用。具体而言，我们提出了一种新颖的交叉模态样本选择方法（CromSS），利用给定地理空间场景的多个传感器/模态d模拟的像素x和类别c之间的类别分布P^{(d)}(x,c)。基于P^{(d)}(x,c)的熵，沟通了传感器d上的预测结果的一致性。通过每个传感器d对噪声类别标签的置信度P^{(d)}(x,c=y(x))确定噪声标签采样。通过对来自全球采样的SSL4EO-S12数据集的Sentinel-1（雷达）和Sentinel-2（光学）卫星图像进行预训练，我们验证了我们方法的性能。对DFC2020数据集上的迁移学习评估（下游任务）证实了该方法在遥感图像分割中的有效性。

CromSS：面向遥感图像分割的带有噪声标签的跨模态预训练