Interpreting the decisions of deep learning models has been actively studied
since the explosion of deep neural networks. One of the most convincing
interpretation approaches is salience-based visual interpretation, such as
Grad-CAM, where the generation of attention maps depends merely on categorical
labels. Although existing interpretation methods can provide explainable
decision clues, they often yield partial correspondence between image and
saliency maps due to the limited discriminative information from one-hot
labels. This paper develops a Language-Image COnsistency model for explainable
image classification, termed LICO, by correlating learnable linguistic prompts
with corresponding visual features in a coarse-to-fine manner. Specifically, we
first establish a coarse global manifold structure alignment by minimizing the
distance between the distributions of image and language features. We then
achieve fine-grained saliency maps by applying optimal transport (OT) theory to
assign local feature maps with class-specific prompts. Extensive experimental
results on eight benchmark datasets demonstrate that the proposed LICO achieves
a significant improvement in generating more explainable attention maps in
conjunction with existing interpretation methods such as Grad-CAM. Remarkably,
LICO improves the classification performance of existing models without
introducing any computational overhead during inference. Source code is made
available at this https URL

通过将可学习的语言提示与相应的视觉特征相关联，以粗到精的方式建立了一种用于可解释图像分类的语言图像一致性模型（LICO），通过最小化图像和语言特征分布之间的距离，建立了粗略的全球流形结构对齐，然后通过应用最优传输（OT）理论将局部特征图与类别特定的提示分配，从而实现细粒度的显著性图。广泛的实验结果表明，LICO 相对于现有的解释方法（如 Grad-CAM）在生成更可解释的注意力图方面取得了显著进展。值得注意的是，在推理期间，LICO 改进了现有模型的分类性能，并未引入任何计算开销。