This report describes the winning solution to the WeatherProof Dataset
Challenge (CVPR 2024 UG2+ Track 3). Details regarding the challenge are
available at this https URL We propose an
enhanced semantic segmentation pipeline for this challenge. Firstly, we improve
semantic segmentation models, using backbone pretrained with Depth Anything to
improve UperNet model and SETRMLA model, and adding language guidance based on
both weather and category information to InternImage model. Secondly, we
introduce a new dataset WeatherProofExtra with wider viewing angle and employ
data augmentation methods, including adverse weather and super-resolution.
Finally, effective training strategies and ensemble method are applied to
improve final performance further. Our solution is ranked 1st on the final
leaderboard. Code will be available at
this https URL

描述第一名解决方案，提出了一种增强的语义分割管道，通过使用经过深度预训练的骨干模型、语言引导和数据增强方法来改进天气下的语义分割模型，并采用有效的训练策略和集成方法来进一步提高性能。

天气固定数据集挑战的增强语义分割流程

Enhanced Semantic Segmentation Pipeline for WeatherProof Dataset  Challenge

We propose a method to infer semantic segmentation maps from images captured
under adverse weather conditions. We begin by examining existing models on
images degraded by weather conditions such as rain, fog, or snow, and found
that they exhibit a large performance drop as compared to those captured under
clear weather. To control for changes in scene structures, we propose
WeatherProof, the first semantic segmentation dataset with accurate clear and
adverse weather image pairs that share an underlying scene. Through this
dataset, we analyze the error modes in existing models and found that they were
sensitive to the highly complex combination of different weather effects
induced on the image during capture. To improve robustness, we propose a way to
use language as guidance by identifying contributions of adverse weather
conditions and injecting that as "side information". Models trained using our
language guidance exhibit performance gains by up to 10.2% in mIoU on
WeatherProof, up to 8.44% in mIoU on the widely used ACDC dataset compared to
standard training techniques, and up to 6.21% in mIoU on the ACDC dataset as
compared to previous SOTA methods.

我们提出了一种从在恶劣天气条件下拍摄的图像中推断语义分割地图的方法，通过利用语言作为指导，在不同的气候效应对图像的影响下，模型在 WeatherProof 和 ACDC 数据集上表现出了更好的性能。

借助语言引导的恶劣天气语义分割

WeatherProof: Leveraging Language Guidance for Semantic Segmentation in  Adverse Weather

The introduction of large, foundational models to computer vision has led to
drastically improved performance on the task of semantic segmentation. However,
these existing methods exhibit a large performance drop when testing on images
degraded by weather conditions such as rain, fog, or snow. We introduce a
general paired-training method that can be applied to all current foundational
model architectures that leads to improved performance on images in adverse
weather conditions. To this end, we create the WeatherProof Dataset, the first
semantic segmentation dataset with accurate clear and adverse weather image
pairs, which not only enables our new training paradigm, but also improves the
evaluation of the performance gap between clear and degraded segmentation. We
find that training on these paired clear and adverse weather frames which share
an underlying scene results in improved performance on adverse weather data.
With this knowledge, we propose a training pipeline which accentuates the
advantages of paired-data training using consistency losses and language
guidance, which leads to performance improvements by up to 18.4% as compared to
standard training procedures.

使用通用的配对训练方法，改进了大型计算机视觉模型在恶劣气候下的语义分割性能，并创建了 WeatherProof 数据集，提供了准确的恶劣气候和晴朗天气的图像对，进一步改善了分割性能的评估，并通过一种采用一致性损失和语言引导的训练流程，使性能提高了 18.4%。