Semantic segmentation of remote sensing images plays a vital role in a wide
range of Earth Observation (EO) applications, such as land use land cover
mapping, environment monitoring, and sustainable development. Driven by rapid
developments in Artificial Intelligence (AI), deep learning (DL) has emerged as
the mainstream tool for semantic segmentation and achieved many breakthroughs
in the field of remote sensing. However, the existing DL-based methods mainly
focus on unimodal visual data while ignoring the rich multimodal information
involved in the real world, usually demonstrating weak reliability and
generlization. Inspired by the success of Vision Transformers and large
language models, we propose a novel metadata-collaborative multimodal
segmentation network (MetaSegNet) that applies vision-language representation
learning for semantic segmentation of remote sensing images. Unlike the common
model structure that only uses unimodal visual data, we extract the key
characteristic (i.e. the climate zone) from freely available remote sensing
image metadata and transfer it into knowledge-based text prompts via the
generic ChatGPT. Then, we construct an image encoder, a text encoder and a
crossmodal attention fusion subnetwork to extract the image and text feature
and apply image-text interaction. Benefiting from such a design, the proposed
MetaSegNet demonstrates superior generalization and achieves competitive
accuracy with state-of-the-art semantic segmentation methods on the large-scale
OpenEarthMap dataset (68.6% mIoU) and Potsdam dataset (93.3% mean F1 score) as
well as LoveDA dataset (52.2% mIoU).

我们提出了一种基于视觉 - 语言表示学习的元数据协作多模态分割网络（MetaSegNet），该网络用于遥感图像的语义分割，并展示了卓越的泛化性能和与现有方法相竞争的准确性。

MetaSegNet: 基于元数据协同的视觉语言表示学习用于遥感图像语义分割

MetaSegNet: Metadata-collaborative Vision-Language Representation  Learning for Semantic Segmentation of Remote Sensing Images

To address the annotation scarcity issue in some cases of semantic
segmentation, there have been a few attempts to develop the segmentation model
in the few-shot learning paradigm. However, most existing methods only focus on
the traditional 1-way segmentation setting (i.e., one image only contains a
single object). This is far away from practical semantic segmentation tasks
where the K-way setting (K>1) is usually required by performing the accurate
multi-object segmentation. To deal with this issue, we formulate the few-shot
semantic segmentation task as a learning-based pixel classification problem and
propose a novel framework called MetaSegNet based on meta-learning. In
MetaSegNet, an architecture of embedding module consisting of the global and
local feature branches is developed to extract the appropriate meta-knowledge
for the few-shot segmentation. Moreover, we incorporate a linear model into
MetaSegNet as a base learner to directly predict the label of each pixel for
the multi-object segmentation. Furthermore, our MetaSegNet can be trained by
the episodic training mechanism in an end-to-end manner from scratch.
Experiments on two popular semantic segmentation datasets, i.e., PASCAL VOC and
COCO, reveal the effectiveness of the proposed MetaSegNet in the K-way few-shot
semantic segmentation task.

该研究提出了一种基于元学习的新框架 MetaSegNet，通过全局和局部特征分支，在元学习中提取少样本语义分割的适当元知识，并将线性模型集成到 MetaSegNet 中，以实现一次性精确地多目标分割。实验结果表明，该方法在 K 路少样本语义分割任务中表现出良好的效果。