Masked image modeling (MIM) is a highly popular and effective self-supervised
learning method for image understanding. Existing MIM-based methods mostly
focus on spatial feature modeling, neglecting spectral feature modeling.
Meanwhile, existing MIM-based methods use Transformer for feature extraction,
some local or high-frequency information may get lost. To this end, we propose
a spatial-spectral masked auto-encoder (SS-MAE) for HSI and LiDAR/SAR data
joint classification. Specifically, SS-MAE consists of a spatial-wise branch
and a spectral-wise branch. The spatial-wise branch masks random patches and
reconstructs missing pixels, while the spectral-wise branch masks random
spectral channels and reconstructs missing channels. Our SS-MAE fully exploits
the spatial and spectral representations of the input data. Furthermore, to
complement local features in the training stage, we add two lightweight CNNs
for feature extraction. Both global and local features are taken into account
for feature modeling. To demonstrate the effectiveness of the proposed SS-MAE,
we conduct extensive experiments on three publicly available datasets.
Extensive experiments on three multi-source datasets verify the superiority of
our SS-MAE compared with several state-of-the-art baselines. The source codes
are available at https://github.com/summitgao/SS-MAE.

提出了一种空间 - 光谱掩码自编码器 (SS-MAE) 用于高光谱成像和激光雷达 / 合成孔径雷达数据的联合分类，实现了空间和光谱表示的充分利用，并通过在训练阶段添加两个轻量级卷积神经网络来补充局部特征，成功在三个公开数据集上验证了 SS-MAE 相对于其他先进基线方法的优越性。