Vision-language models enable open-world classification of objects without
the need for any retraining. While this zero-shot paradigm marks a significant
advance, even today's best models exhibit skewed performance when objects are
dissimilar from their typical depiction. Real world objects such as pears
appear in a variety of forms -- from diced to whole, on a table or in a bowl --
yet standard VLM classifiers map all instances of a class to a \it{single
vector based on the class label}. We argue that to represent this rich
diversity within a class, zero-shot classification should move beyond a single
vector. We propose a method to encode and account for diversity within a class
using inferred attributes, still in the zero-shot setting without retraining.
We find our method consistently outperforms standard zero-shot classification
over a large suite of datasets encompassing hierarchies, diverse object states,
and real-world geographic diversity, as well finer-grained datasets where
intra-class diversity may be less prevalent. Importantly, our method is
inherently interpretable, offering faithful explanations for each inference to
facilitate model debugging and enhance transparency. We also find our method
scales efficiently to a large number of attributes to account for diversity --
leading to more accurate predictions for atypical instances. Finally, we
characterize a principled trade-off between overall and worst class accuracy,
which can be tuned via a hyperparameter of our method. We hope this work spurs
further research into the promise of zero-shot classification beyond a single
class vector for capturing diversity in the world, and building transparent AI
systems without compromising performance.

用推理属性在不需要重新训练的零 - shot 分类环境下对类别内的多样性进行编码和计算，提高零 - shot 分类的准确性、可解释性，以及适应性，从而实现透明的 AI 系统。

拥抱多样性：超过每个类别一个向量的可解释零 - shot 分类

Embracing Diversity: Interpretable Zero-shot classification beyond one  vector per class

A central goal of explainable artificial intelligence (XAI) is to improve the
trust relationship in human-AI interaction. One assumption underlying research
in transparent AI systems is that explanations help to better assess
predictions of machine learning (ML) models, for instance by enabling humans to
identify wrong predictions more efficiently. Recent empirical evidence however
shows that explanations can have the opposite effect: When presenting
explanations of ML predictions humans often tend to trust ML predictions even
when these are wrong. Experimental evidence suggests that this effect can be
attributed to how intuitive, or human, an AI or explanation appears. This
effect challenges the very goal of XAI and implies that responsible usage of
transparent AI methods has to consider the ability of humans to distinguish
machine generated from human explanations. Here we propose a quantitative
metric for XAI methods based on Turing's imitation game, a Turing Test for
Transparency. A human interrogator is asked to judge whether an explanation was
generated by a human or by an XAI method. Explanations of XAI methods that can
not be detected by humans above chance performance in this binary
classification task are passing the test. Detecting such explanations is a
requirement for assessing and calibrating the trust relationship in human-AI
interaction. We present experimental results on a crowd-sourced text
classification task demonstrating that even for basic ML models and XAI
approaches most participants were not able to differentiate human from machine
generated explanations. We discuss ethical and practical implications of our
results for applications of transparent ML.

该研究旨在通过构建透明的 AI 系统来提高人机交互的信任关系，并通过引入图灵测试来建立可信度评估体系，以便解释机器学习模型的结果，证明透明的 ML 方法能够通过人机交互获得信任，但也需要注意到人类与机器的辨别能力。