The advent of high-resolution multispectral/hyperspectral sensors, LiDAR DSM
(Digital Surface Model) information and many others has provided us with an
unprecedented wealth of data for Earth Observation. Multimodal AI seeks to
exploit those complementary data sources, particularly for complex tasks like
semantic segmentation. While specialized architectures have been developed,
they are highly complicated via significant effort in model design, and require
considerable re-engineering whenever a new modality emerges. Recent trends in
general-purpose multimodal networks have shown great potential to achieve
state-of-the-art performance across multiple multimodal tasks with one unified
architecture. In this work, we investigate the performance of PerceiverIO, one
in the general-purpose multimodal family, in the remote sensing semantic
segmentation domain. Our experiments reveal that this ostensibly universal
network struggles with object scale variation in remote sensing images and
fails to detect the presence of cars from a top-down view. To address these
issues, even with extreme class imbalance issues, we propose a spatial and
volumetric learning component. Specifically, we design a UNet-inspired module
that employs 3D convolution to encode vital local information and learn
cross-modal features simultaneously, while reducing network computational
burden via the cross-attention mechanism of PerceiverIO. The effectiveness of
the proposed component is validated through extensive experiments comparing it
with other methods such as 2D convolution, and dual local module (\ie the
combination of Conv2D 1x1 and Conv2D 3x3 inspired by UNetFormer). The proposed
method achieves competitive results with specialized architectures like
UNetFormer and SwinUNet, showing its potential to minimize network architecture
engineering with a minimal compromise on the performance.

在遥感语义分割领域中，本研究调查了 PerceiverIO 等普适性多模态网络在处理遥感图像中的物体尺度变化和车辆检测方面的表现，并提出一种空间和体积学习组件来解决这些问题。通过广泛的实验证明，该方法在与 UNetFormer 和 SwinUNet 等专用架构相比具有竞争力的结果，显示出最小化网络架构工程且性能不受明显影响的潜力。