Despite the rapid evolution of semantic segmentation for land cover
classification in high-resolution remote sensing imagery, integrating multiple
data modalities such as Digital Surface Model (DSM), RGB, and Near-infrared
(NIR) remains a challenge. Current methods often process only two types of
data, missing out on the rich information that additional modalities can
provide. Addressing this gap, we propose a novel \textbf{L}ightweight
\textbf{M}ultimodal data \textbf{F}usion \textbf{Net}work (LMFNet) to
accomplish the tasks of fusion and semantic segmentation of multimodal remote
sensing images. LMFNet uniquely accommodates various data types simultaneously,
including RGB, NirRG, and DSM, through a weight-sharing, multi-branch vision
transformer that minimizes parameter count while ensuring robust feature
extraction. Our proposed multimodal fusion module integrates a
\textit{Multimodal Feature Fusion Reconstruction Layer} and \textit{Multimodal
Feature Self-Attention Fusion Layer}, which can reconstruct and fuse multimodal
features. Extensive testing on public datasets such as US3D, ISPRS Potsdam, and
ISPRS Vaihingen demonstrates the effectiveness of LMFNet. Specifically, it
achieves a mean Intersection over Union ($mIoU$) of 85.09\% on the US3D
dataset, marking a significant improvement over existing methods. Compared to
unimodal approaches, LMFNet shows a 10\% enhancement in $mIoU$ with only a 0.5M
increase in parameter count. Furthermore, against bimodal methods, our approach
with trilateral inputs enhances $mIoU$ by 0.46 percentage points.

提出了一种轻量级多模态数据融合网络（LMFNet），用于实现多模态遥感图像的融合和语义分割，通过权重共享、多分支视觉变换来同时容纳 RGB、NirRG 和 DSM 等各种数据类型，并通过多模态特征融合重构层和多模态特征自注意融合层对多模态特征进行重构和融合，实验证明了 LMFNet 的有效性。