Multi-modal semantic segmentation significantly enhances AI agents'
perception and scene understanding, especially under adverse conditions like
low-light or overexposed environments. Leveraging additional modalities
(X-modality) like thermal and depth alongside traditional RGB provides
complementary information, enabling more robust and reliable segmentation. In
this work, we introduce Sigma, a Siamese Mamba network for multi-modal semantic
segmentation, utilizing the Selective Structured State Space Model, Mamba.
Unlike conventional methods that rely on CNNs, with their limited local
receptive fields, or Vision Transformers (ViTs), which offer global receptive
fields at the cost of quadratic complexity, our model achieves global receptive
fields coverage with linear complexity. By employing a Siamese encoder and
innovating a Mamba fusion mechanism, we effectively select essential
information from different modalities. A decoder is then developed to enhance
the channel-wise modeling ability of the model. Our method, Sigma, is
rigorously evaluated on both RGB-Thermal and RGB-Depth segmentation tasks,
demonstrating its superiority and marking the first successful application of
State Space Models (SSMs) in multi-modal perception tasks. Code is available at
this https URL

该研究介绍了 Sigma，一种用于多模态语义分割的 Siamese Mamba 网络，利用选择性结构状态空间模型（Mamba），并在 RGB - 热力图和 RGB - 深度分割任务上进行了评估，展示了其在多模态感知任务中的优越性和标志性应用

Sigma: 双子曼巴网络用于多模态语义分割

Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation

Geometric information in the normalized digital surface models (nDSM) is
highly correlated with the semantic class of the land cover. Exploiting two
modalities (RGB and nDSM (height)) jointly has great potential to improve the
segmentation performance. However, it is still an under-explored field in
remote sensing due to the following challenges. First, the scales of existing
datasets are relatively small and the diversity of existing datasets is
limited, which restricts the ability of validation. Second, there is a lack of
unified benchmarks for performance assessment, which leads to difficulties in
comparing the effectiveness of different models. Last, sophisticated
multi-modal semantic segmentation methods have not been deeply explored for
remote sensing data. To cope with these challenges, in this paper, we introduce
a new remote-sensing benchmark dataset for multi-modal semantic segmentation
based on RGB-Height (RGB-H) data. Towards a fair and comprehensive analysis of
existing methods, the proposed benchmark consists of 1) a large-scale dataset
including co-registered RGB and nDSM pairs and pixel-wise semantic labels; 2) a
comprehensive evaluation and analysis of existing multi-modal fusion strategies
for both convolutional and Transformer-based networks on remote sensing data.
Furthermore, we propose a novel and effective Transformer-based intermediary
multi-modal fusion (TIMF) module to improve the semantic segmentation
performance through adaptive token-level multi-modal fusion.The designed
benchmark can foster future research on developing new methods for multi-modal
learning on remote sensing data. Extensive analyses of those methods are
conducted and valuable insights are provided through the experimental results.
Code for the benchmark and baselines can be accessed at
https://github.com/EarthNets/RSI-MMSegmentation.

本文提出一种基于 RGB-Height 数据的遥感多模态语义分割新基准数据集，包含大规模数据集，认真评测已有的方法并提出一种新的 Transformer-based 中间多模态融合 (TIMF) 模块来适应令人满意的语义分割性能。

GAMUS：一种适用于遥感数据的几何感知多模态语义分割基准测试

GAMUS: A Geometry-aware Multi-modal Semantic Segmentation Benchmark for  Remote Sensing Data

Computer vision algorithms have been prevalently utilized for 3-D road
imaging and pothole detection for over two decades. Nonetheless, there is a
lack of systematic survey articles on state-of-the-art (SoTA) computer vision
techniques, especially deep learning models, developed to tackle these
problems. This article first introduces the sensing systems employed for 2-D
and 3-D road data acquisition, including camera(s), laser scanners, and
Microsoft Kinect. Afterward, it thoroughly and comprehensively reviews the SoTA
computer vision algorithms, including (1) classical 2-D image processing, (2)
3-D point cloud modeling and segmentation, and (3) machine/deep learning,
developed for road pothole detection. This article also discusses the existing
challenges and future development trends of computer vision-based road pothole
detection approaches: classical 2-D image processing-based and 3-D point cloud
modeling and segmentation-based approaches have already become history; and
Convolutional neural networks (CNNs) have demonstrated compelling road pothole
detection results and are promising to break the bottleneck with the future
advances in self/un-supervised learning for multi-modal semantic segmentation.
We believe that this survey can serve as practical guidance for developing the
next-generation road condition assessment systems.

本文介绍了二 / 三维道路数据获取的传感系统（相机、激光扫描仪和微软 Kinect），并全面综述了用于道路坑洞检测的最先进计算机视觉算法，包括传统的二维图像处理、三维点云建模和分割、以及机器 / 深度学习算法，同时讨论了现有挑战和未来发展趋势。