Leveraging information across diverse modalities is known to enhance
performance on multimodal segmentation tasks. However, effectively fusing
information from different modalities remains challenging due to the unique
characteristics of each modality. In this paper, we propose a novel fusion
strategy that can effectively fuse information from different combinations of
four different modalities: RGB, Angle of Linear Polarization (AoLP), Degree of
Linear Polarization (DoLP) and Near-Infrared (NIR). We also propose a new model
named Multi-Modal Segmentation Transformer (MMSFormer) that incorporates the
proposed fusion strategy to perform multimodal material segmentation. MMSFormer
achieves 52.05% mIoU outperforming the current state-of-the-art on Multimodal
Material Segmentation (MCubeS) dataset. For instance, our method provides
significant improvement in detecting gravel (+10.4%) and human (+9.1%) classes.
Ablation studies show that different modules in the fusion block are crucial
for overall model performance. Furthermore, our ablation studies also highlight
the capacity of different input modalities to improve performance in the
identification of different types of materials. The code and pretrained models
will be made available at this https URL

提出了一种新颖的融合策略，能够有效地融合来自四种不同模态（RGB、AoLP、DoLP 和 NIR）的信息，结合提出的融合策略，使用新模型 Multi-Modal Segmentation Transformer（MMSFormer）在 MCubeS 数据集上实现了 52.05% 的 mIoU，并且在检测砾石（+10.4%）和人物（+9.1%）类别方面提供了显著的改进。

多模态变换器用于材料分割

Multimodal Transformer for Material Segmentation

Achieving accurate material segmentation for 3-channel RGB images is
challenging due to the considerable variation in a material's appearance.
Hyperspectral images, which are sets of spectral measurements sampled at
multiple wavelengths, theoretically offer distinct information for material
identification, as variations in intensity of electromagnetic radiation
reflected by a surface depend on the material composition of a scene. However,
existing hyperspectral datasets are impoverished regarding the number of images
and material categories for the dense material segmentation task, and
collecting and annotating hyperspectral images with a spectral camera is
prohibitively expensive. To address this, we propose a new model, the
MatSpectNet to segment materials with recovered hyperspectral images from RGB
images. The network leverages the principles of colour perception in modern
cameras to constrain the reconstructed hyperspectral images and employs the
domain adaptation method to generalise the hyperspectral reconstruction
capability from a spectral recovery dataset to material segmentation datasets.
The reconstructed hyperspectral images are further filtered using learned
response curves and enhanced with human perception. The performance of
MatSpectNet is evaluated on the LMD dataset as well as the OpenSurfaces
dataset. Our experiments demonstrate that MatSpectNet attains a 1.60% increase
in average pixel accuracy and a 3.42% improvement in mean class accuracy
compared with the most recent publication. The project code is attached to the
supplementary material and will be published on GitHub.

提出了一种新模型 MatSpectNet，利用从 RGB 图像中恢复的高光谱图像来进行物质分割，通过色彩感知原理和域自适应方法，将高光谱图像的恢复能力推广到物质分割数据集中，实验证明 MatSpectNet 在像素准确性和类别准确性方面比现有方法提高了 1.60% 和 3.42%。