The application of zero-shot learning in computer vision has been
revolutionized by the use of image-text matching models. The most notable
example, CLIP, has been widely used for both zero-shot classification and
guiding generative models with a text prompt. However, the zero-shot use of
CLIP is unstable with respect to the phrasing of the input text, making it
necessary to carefully engineer the prompts used. We find that this instability
stems from a selective similarity score, which is based only on a subset of the
semantically meaningful input tokens. To mitigate it, we present a novel
explainability-based approach, which adds a loss term to ensure that CLIP
focuses on all relevant semantic parts of the input, in addition to employing
the CLIP similarity loss used in previous works. When applied to one-shot
classification through prompt engineering, our method yields an improvement in
the recognition rate, without additional training or fine-tuning. Additionally,
we show that CLIP guidance of generative models using our method significantly
improves the generated images. Finally, we demonstrate a novel use of CLIP
guidance for text-based image generation with spatial conditioning on object
location, by requiring the image explainability heatmap for each object to be
confined to a pre-determined bounding box.

本研究提出了一种基于解释性的方法来解决在零样本学习和图像生成中使用 CLIP 时输入文本的稳定性问题，此方法通过增加一项损失项来确保 CLIP 关注所有相关的语义部分，并且可以提高图像识别率和生成图像的质量。同时，研究还展示了 CLIP 在一次性分类、对生成模型进行指导和有空间条件的基于文本的图像生成方面的新型应用。