Knowing when a trained segmentation model is encountering data that is
different to its training data is important. Understanding and mitigating the
effects of this play an important part in their application from a performance
and assurance perspective - this being a safety concern in applications such as
autonomous vehicles (AVs). This work presents a segmentation network that can
detect errors caused by challenging test domains without any additional
annotation in a single forward pass. As annotation costs limit the diversity of
labelled datasets, we use easy-to-obtain, uncurated and unlabelled data to
learn to perform uncertainty estimation by selectively enforcing consistency
over data augmentation. To this end, a novel segmentation benchmark based on
the SAX Dataset is used, which includes labelled test data spanning three
autonomous-driving domains, ranging in appearance from dense urban to off-road.
The proposed method, named Gamma-SSL, consistently outperforms uncertainty
estimation and Out-of-Distribution (OoD) techniques on this difficult benchmark
- by up to 10.7% in area under the receiver operating characteristic (ROC)
curve and 19.2% in area under the precision-recall (PR) curve in the most
challenging of the three scenarios.

基于未标记数据的 Gamma-SSL 方法通过选择性强制一致性来进行不确定性估计，用于在自动驾驶环境中检测语义分割模型的错误。在具有挑战性的分类基准上，该方法在 ROC 曲线下面积和 PR 曲线下面积方面的表现超过了不确定性估计和离群分布技术。

通过未标记数据的不确定性估计缓解语义分割中的分布偏移

Mitigating Distributional Shift in Semantic Segmentation via Uncertainty  Estimation from Unlabelled Data

While instruction-tuned models have shown remarkable success in various
natural language processing tasks, accurately evaluating their ability to
follow instructions remains challenging. Existing benchmarks primarily focus on
common instructions that align well with what the model learned during
training. However, proficiency in responding to these instructions does not
necessarily imply strong ability in instruction following. In this paper, we
propose a novel instruction-following evaluation protocol called verbalizer
manipulation. It instructs the model to verbalize the task label with words
aligning with model priors to different extents, adopting verbalizers from
highly aligned (e.g., outputting ``postive'' for positive sentiment), to
minimally aligned (e.g., outputting ``negative'' for positive sentiment).
Verbalizer manipulation can be seamlessly integrated with any classification
benchmark to examine the model's reliance on priors and its ability to override
them to accurately follow the instructions. We conduct a comprehensive
evaluation of four major model families across nine datasets, employing twelve
sets of verbalizers for each of them. We observe that the instruction-following
abilities of models, across different families and scales, are significantly
distinguished by their performance on less natural verbalizers. Even the
strongest GPT-4 model struggles to perform better than random guessing on the
most challenging verbalizer, emphasizing the need for continued advancements to
improve their instruction-following abilities.

我们提出了一种名为 verbalizer manipulation 的新型指令跟踪评估协议，该协议通过使用高度匹配到轻微匹配的语言转化器，结合任何分类基准来检查模型对先验知识的依赖程度以及其是否能准确遵循指令。我们对四个重要的模型系列进行了全面评估，使用了每个系列的十二组语言转化器，发现模型在不同系列和规模上的指令遵循能力在较不自然的语言转化器上有明显差异。即使是最强大的 GPT-4 模型在最具挑战性的语言转化器上也难以比随机猜测更好，强调了需要持续进步来改善其指令遵循能力。