Unsupervised pre-training methods for large vision models have shown to
enhance performance on downstream supervised tasks. Developing similar
techniques for satellite imagery presents significant opportunities as
unlabelled data is plentiful and the inherent temporal and multi-spectral
structure provides avenues to further improve existing pre-training strategies.
In this paper, we present SatMAE, a pre-training framework for temporal or
multi-spectral satellite imagery based on Masked Autoencoder (MAE). To leverage
temporal information, we include a temporal embedding along with independently
masking image patches across time. In addition, we demonstrate that encoding
multi-spectral data as groups of bands with distinct spectral positional
encodings is beneficial. Our approach yields strong improvements over previous
state-of-the-art techniques, both in terms of supervised learning performance
on benchmark datasets (up to $\uparrow$ 7%), and transfer learning performance
on downstream remote sensing tasks, including land cover classification (up to
$\uparrow$ 14%) and semantic segmentation. Code and data are available on the
project website: this https URL

通过使用遮蔽自动编码器（MAE）的卫星图像的预训练框架 SatMAE，我们结合时间嵌入和分组打包多光谱数据以提高预训练能力，进一步提高基准数据集上的监督学习性能（最高可达 7%），并在后续的遥感分类任务中表现出强大的转移学习能力（最高可达 14%）。