Using images acquired by different satellite sensors has shown to improve
classification performance in the framework of crop mapping from satellite
image time series (SITS). Existing state-of-the-art architectures use
self-attention mechanisms to process the temporal dimension and convolutions
for the spatial dimension of SITS. Motivated by the success of purely
attention-based architectures in crop mapping from single-modal SITS, we
introduce several multi-modal multi-temporal transformer-based architectures.
Specifically, we investigate the effectiveness of Early Fusion, Cross Attention
Fusion and Synchronized Class Token Fusion within the Temporo-Spatial Vision
Transformer (TSViT). Experimental results demonstrate significant improvements
over state-of-the-art architectures with both convolutional and self-attention
components.

通过使用来自不同卫星传感器获取的图像，已经证实在卫星图像时间序列（SITS）的作物分布图框架中，可以提高分类性能。现有的最新架构使用自注意机制处理时间维度和卷积处理空间维度。受到单模态 SITS 作物分布图中纯注意力架构的成功启发，我们引入了几种多模态多时序变换器架构。具体来说，我们研究了在时间空间视觉变换器（TSViT）中早期融合、交叉注意融合和同步类标记融合的有效性。实验结果表明，相较于具有卷积和自注意组件的最新架构，我们的架构显著改善了性能。

卫星影像时序的多模态视觉转换器用于作物分类

Multi-Modal Vision Transformers for Crop Mapping from Satellite Image  Time Series

Most current NeRF variants for satellites are designed for one specific scene
and fall short of generalization to new geometry. Additionally, the RGB images
require pan-sharpening as an independent preprocessing step. This paper
introduces psPRF, a Planar Neural Radiance Field designed for paired
low-resolution RGB (LR-RGB) and high-resolution panchromatic (HR-PAN) images
from satellite sensors with Rational Polynomial Cameras (RPC). To capture the
cross-modal prior from both of the LR-RGB and HR-PAN images, for the
Unet-shaped architecture, we adapt the encoder with explicit
spectral-to-spatial convolution (SSConv) to enhance the multimodal
representation ability. To support the generalization ability of psRPF across
scenes, we adopt projection loss to ensure strong geometry self-supervision.
The proposed method is evaluated with the multi-scene WorldView-3 LR-RGB and
HR-PAN pairs, and achieves state-of-the-art performance.

该论文介绍了一种针对卫星传感器的低分辨率 RGB 图像和高分辨率全色图像的 Planar 神经辐射场（psPRF），通过采用显式的光谱到空间卷积（SSConv）来增强多模态表示能力，并采用投影损失来支持 psPRF 的泛化能力，该方法在多场景的 WorldView-3 数据集上取得了最先进的性能。