This paper proposes a novel multi-temporal urban mapping approach using
multi-modal satellite data from the Sentinel-1 Synthetic Aperture Radar (SAR)
and Sentinel-2 MultiSpectral Instrument (MSI) missions. In particular, it
focuses on the problem of a partly missing optical modality due to clouds. The
proposed model utilizes two networks to extract features from each modality
separately. In addition, a reconstruction network is utilized to approximate
the optical features based on the SAR data in case of a missing optical
modality. Our experiments on a multi-temporal urban mapping dataset with
Sentinel-1 SAR and Sentinel-2 MSI data demonstrate that the proposed method
outperforms a multi-modal approach that uses zero values as a replacement for
missing optical data, as well as a uni-modal SAR-based approach. Therefore, the
proposed method is effective in exploiting multi-modal data, if available, but
it also retains its effectiveness in case the optical modality is missing.

该研究提出了一种基于多模态卫星数据的新型多时相城市制图方法，其中考虑到了部分缺失的光学模态，所提出的模型采用了两个网络来分别提取每个模态的特征，并利用重建网络根据 SAR 数据近似光学特征并保持其数据的有效性。结果表明该方法在使用 SAR 数据以补充缺失光学数据方面的实用性较强，可有效开发和利用多模态数据。

部分失效光学模态的多时相城市制图的多模态深度学习

Multi-Modal Deep Learning for Multi-Temporal Urban Mapping With a Partly  Missing Optical Modality

Earth observation satellites have been continuously monitoring the earth
environment for years at different locations and spectral bands with different
modalities. Due to complex satellite sensing conditions (e.g., weather, cloud,
atmosphere, orbit), some observations for certain modalities, bands, locations,
and times may not be available. The MultiEarth Matrix Completion Challenge in
CVPR 2022 [1] provides the multimodal satellite data for addressing such data
sparsity challenges with the Amazon Rainforest as the region of interest. This
work proposes an adaptive real-time multimodal regression and generation
framework and achieves superior performance on unseen test queries in this
challenge with an LPIPS of 0.2226, a PSNR of 123.0372, and an SSIM of 0.6347.

本文提出了一种适应性实时多模态回归和生成框架来解决卫星数据稀疏性的挑战，并在 CVPR 2022 的 MultiEarth 矩阵完备性挑战中取得了优异的性能，其中包括 LPIPS 为 0.2226，PSNR 为 123.0372 和 SSIM 为 0.6347。