Large multimodal models such as Stable Diffusion can generate, detect, and
classify new visual concepts after fine-tuning just a single word embedding. Do
models learn similar words for the same concepts (i.e. <orange-cat> = orange +
cat)? We conduct a large-scale analysis on three state-of-the-art models in
text-to-image generation, open-set object detection, and zero-shot
classification, and find that new word embeddings are model-specific and
non-transferable. Across 4,800 new embeddings trained for 40 diverse visual
concepts on four standard datasets, we find perturbations within an
$\epsilon$-ball to any prior embedding that generate, detect, and classify an
arbitrary concept. When these new embeddings are spliced into new models,
fine-tuning that targets the original model is lost. We show popular soft
prompt-tuning approaches find these perturbative solutions when applied to
visual concept learning tasks, and embeddings for visual concepts are not
transferable. Code for reproducing our work is available at:
this https URL

大型多模态模型通过仅微调单个词嵌入就能生成、检测和分类新的视觉概念，但我们发现模型学习相似的词语表示同一概念的能力是模型特定且不可转移的。我们对三种先进模型在文本到图像生成、开放集目标检测和零样本分类领域进行了大规模分析，发现新的词嵌入是模型特定且不可转移的。我们在四个标准数据集上针对 40 个不同的视觉概念训练了 4800 个新的嵌入，发现在一个 epsilon 球内的任何先前嵌入的扰动都能生成、检测和分类任意的概念。当这些新的词嵌入被插入新模型时，针对原始模型的微调将失效。我们展示了流行的软提示微调方法在视觉概念学习任务中发现这些扰动解，而视觉概念的嵌入是不可转移的。可复现我们工作的代码可在此 https 网址找到。

跨模型理解视觉概念

Understanding Visual Concepts Across Models

Due to the increasing use of machine learning in practice it becomes more and
more important to be able to explain the prediction and behavior of machine
learning models. An instance of explanations are counterfactual explanations
which provide an intuitive and useful explanations of machine learning models.
In this survey we review model-specific methods for efficiently computing
counterfactual explanations of many different machine learning models and
propose methods for models that have not been considered in literature so far.

机器学习模型日益广泛应用，因此解释其预测和行为变得越来越重要。本文综述了针对多种不同机器学习模型高效计算对抗性解释的模型特定方法，并提出了文献中尚未考虑的模型方法。