Zero-shot anomaly segmentation using pre-trained foundation models is a
promising approach that enables effective algorithms without expensive,
domain-specific training or fine-tuning. Ensuring that these methods work
across various environmental conditions and are robust to distribution shifts
is an open problem. We investigate the performance of WinCLIP [14] zero-shot
anomaly segmentation algorithm by perturbing test data using three semantic
transformations: bounded angular rotations, bounded saturation shifts, and hue
shifts. We empirically measure a lower performance bound by aggregating across
per-sample worst-case perturbations and find that average performance drops by
up to 20% in area under the ROC curve and 40% in area under the per-region
overlap curve. We find that performance is consistently lowered on three CLIP
backbones, regardless of model architecture or learning objective,
demonstrating a need for careful performance evaluation.

使用预训练的基础模型进行零样本异常分割是一种有前途的方法，它可以在不需要昂贵的领域特定训练或微调的情况下实现有效的算法。我们通过扰动测试数据使用三种语义变换（有界角度旋转、有界饱和度变化和色调变化）来研究 WinCLIP [14] 零样本异常分割算法的性能。通过在每个样本的最坏情况扰动中进行聚合，我们经验性地测量了一个较低的性能下界，并发现平均性能在 ROC 曲线下面积和区域重叠曲线下面积方面下降了最高达 20% 和 40%。我们发现，无论模型架构或学习目标如何，这三种 CLIP 主干的性能都普遍降低，这表明需要进行仔细的性能评估。

基于 CLIP 的零样本异常分割的语义鲁棒性研究

Investigating the Semantic Robustness of CLIP-based Zero-Shot Anomaly  Segmentation

Recently, foundational models such as CLIP and SAM have shown promising
performance for the task of Zero-Shot Anomaly Segmentation (ZSAS). However,
either CLIP-based or SAM-based ZSAS methods still suffer from non-negligible
key drawbacks: 1) CLIP primarily focuses on global feature alignment across
different inputs, leading to imprecise segmentation of local anomalous parts;
2) SAM tends to generate numerous redundant masks without proper prompt
constraints, resulting in complex post-processing requirements. In this work,
we innovatively propose a CLIP and SAM collaboration framework called ClipSAM
for ZSAS. The insight behind ClipSAM is to employ CLIP's semantic understanding
capability for anomaly localization and rough segmentation, which is further
used as the prompt constraints for SAM to refine the anomaly segmentation
results. In details, we introduce a crucial Unified Multi-scale Cross-modal
Interaction (UMCI) module for interacting language with visual features at
multiple scales of CLIP to reason anomaly positions. Then, we design a novel
Multi-level Mask Refinement (MMR) module, which utilizes the positional
information as multi-level prompts for SAM to acquire hierarchical levels of
masks and merges them. Extensive experiments validate the effectiveness of our
approach, achieving the optimal segmentation performance on the MVTec-AD and
VisA datasets.

CLIP 和 SAM 协作的 ClipSAM 框架可用于零样本异常分割任务，通过结合 CLIP 的语义理解和 SAM 的细分结果优化，获得最佳的分割性能。