A vital and rapidly growing application, remote sensing offers vast yet sparsely labeled, spatially aligned multimodal data; this makes self-supervised learning algorithms invaluable. We present CROMA: a framework that combines contrastive and reconstruction self-supervised objectives to learn rich unimodal and multimodal representations. Our method separately encodes masked-out multispectral optical and synthetic aperture radar samples -- aligned in space and time -- and performs cross-modal contrastive learning. Another encoder fuses these sensors, producing joint multimodal encodings that are used to predict the masked patches via a lightweight decoder. We show that these objectives are complementary when leveraged on spatially aligned multimodal data. We also introduce X- and 2D-ALiBi, which spatially biases our cross- and self-attention matrices. These strategies improve representations and allow our models to effectively extrapolate to images up to 17.6x larger at test-time. CROMA outperforms the current SoTA multispectral model, evaluated on: four classification benchmarks -- finetuning (avg. 1.8%), linear (avg. 2.4%) and nonlinear (avg. 1.4%) probing, kNN classification (avg. 3.5%), and K-means clustering (avg. 8.4%); and three segmentation benchmarks (avg. 6.4%). CROMA's rich, optionally multimodal representations can be widely leveraged across remote sensing applications.

CROMA是一个结合对比学习和重构自监督目标的框架，用于学习丰富的单模态和多模态表示；该方法通过分别编码遮蔽的多光谱光学和合成孔径雷达样本，并进行跨模态对比学习，然后将这些传感器融合到一起，生成用于预测遮蔽补丁的联合多模态编码，从而提高了表示，并使模型能够有效地对测试时尺寸达到17.6倍更大的图像进行外推。CROMA在遥感应用中的多个基准测试中表现优于当前的多光谱模型。

CROMA：基于对比雷达光学遮蔽自编码器的遥感表征