A key benefit of deep vision-language models such as CLIP is that they enable zero-shot open vocabulary classification; the user has the ability to define novel class labels via natural language prompts at inference time. However, while CLIP-based zero-shot classifiers have demonstrated competitive performance across a range of domain shifts, they remain highly vulnerable to adversarial attacks. Therefore, ensuring the robustness of such models is crucial for their reliable deployment in the wild. In this work, we introduce Open Vocabulary Certification (OVC), a fast certification method designed for open-vocabulary models like CLIP via randomized smoothing techniques. Given a base "training" set of prompts and their corresponding certified CLIP classifiers, OVC relies on the observation that a classifier with a novel prompt can be viewed as a perturbed version of nearby classifiers in the base training set. Therefore, OVC can rapidly certify the novel classifier using a variation of incremental randomized smoothing. By using a caching trick, we achieve approximately two orders of magnitude acceleration in the certification process for novel prompts. To achieve further (heuristic) speedups, OVC approximates the embedding space at a given input using a multivariate normal distribution bypassing the need for sampling via forward passes through the vision backbone. We demonstrate the effectiveness of OVC on through experimental evaluation using multiple vision-language backbones on the CIFAR-10 and ImageNet test datasets.

在本论文中，我们介绍了一种针对基于开放词汇的模型（如CLIP）的快速认证方法，通过随机平滑技术实现。该方法利用基础“训练”集和对应的认证CLIP分类器，通过将具有新提示的分类器视为基础训练集中附近分类器的扰动版本，从而快速认证新分类器。通过使用缓存技巧，我们在新提示的认证过程中实现了近两个数量级的加速。为了进一步加快速度（启发式），OVC透过使用多元正态分布来近似给定输入的嵌入空间，绕过通过视觉主干进行采样的需要。我们通过对CIFAR-10和ImageNet测试数据集上的多个视觉语言主干进行实验评估，证明了OVC的有效性。

使用增量随机平滑快速验证视觉-语言模型