Utilizing potent representations of the large vision-language models (VLMs)
to accomplish various downstream tasks has attracted increasing attention.
Within this research field, soft prompt learning has become a representative
approach for efficiently adapting VLMs such as CLIP, to tasks like image
classification. However, most existing prompt learning methods learn text
tokens that are unexplainable, which cannot satisfy the stringent
interpretability requirements of Explainable Artificial Intelligence (XAI) in
high-stakes scenarios like healthcare. To address this issue, we propose a
novel explainable prompt learning framework that leverages medical knowledge by
aligning the semantics of images, learnable prompts, and clinical
concept-driven prompts at multiple granularities. Moreover, our framework
addresses the lack of valuable concept annotations by eliciting knowledge from
large language models and offers both visual and textual explanations for the
prompts. Extensive experiments and explainability analyses conducted on various
datasets, with and without concept labels, demonstrate that our method
simultaneously achieves superior diagnostic performance, flexibility, and
interpretability, shedding light on the effectiveness of foundation models in
facilitating XAI. The code will be made publically available.

利用强大的视觉 - 语言模型（VLM）来完成各种下游任务已引起越来越多的关注。我们提出了一种新颖的可解释提示学习框架，通过在多个粒度上对医学知识、可以学习的提示和临床概念驱动的提示的语义进行对齐，来解决这个问题，并为提示提供了视觉和文本解释。通过对各种数据集进行广泛的实验证明，我们的方法同时实现了优越的诊断性能、灵活性和可解释性，为基础模型在促进可解释的人工智能方面的有效性提供了新的视角。

XCoOp：基于概念引导的上下文优化的计算机辅助诊断可解释式提示学习

XCoOp: Explainable Prompt Learning for Computer-Aided Diagnosis via  Concept-guided Context Optimization

This paper is on soft prompt learning for Vision \& Language (V&L) models.
Similarly to their NLP counterparts, V\&L models can be adapted to a downstream
task by learning soft continuous prompts using a few training examples. Current
methods learn the soft prompts by minimizing a cross-entropy loss using as
class weights the features obtained by passing the prompts plus the class names
through the text encoder. Such methods, however, significantly overfit the
training data suffering from large accuracy degradation when tested on unseen
classes from the same domain. Our main contribution, in this paper, is a
surprisingly simple approach to alleviate this problem: we use a second cross
entropy loss to minimize the distance between the learned soft prompts and a
set of hand-engineered manual prompts (obtained by prompt engineering). The
proposed loss can be interpreted in multiple ways including as a regularizer,
as a means for language-based augmentation, and as a way of learning more
discriminative class centroids. Importantly, our formulation is inherently
amenable to including, during training, virtual classes, i.e. class names for
which no visual samples are available, further increasing the robustness of the
learned prompts. Through extensive evaluations on 11 datasets, we show that our
approach (a) significantly outperforms all prior works on soft prompting, and
(b) matches and surpasses, for the first time, the accuracy on novel classes
obtained by hand-crafted prompts and CLIP for the majority of the test
datasets. Code will be made available.

本文介绍了一种软提示学习的方法，用于 Vision & Language 模型。该方法通过使用第二交叉熵损失来最小化学习的软提示与手工工程提示之间的距离，进而提高模型性能，并能够训练虚拟类。在 11 个数据集上进行的广泛评估表明，该方法显著优于所有先前的软提示工作，并在大多数测试数据集上匹配和超越手工制作提示和 CLIP 的新类准确性。