CLIP yielded impressive results on zero-shot transfer learning tasks and is
considered as a foundation model like BERT or GPT3. CLIP vision models that
have a rich representation are pre-trained using the InfoNCE objective and
natural language supervision before they are fine-tuned on particular tasks.
Though CLIP excels at zero-shot transfer learning, it suffers from an
explaining away problem, that is, it focuses on one or few features, while
neglecting other relevant features. This problem is caused by insufficiently
extracting the covariance structure in the original multi-modal data. We
suggest to use modern Hopfield networks to tackle the problem of explaining
away. Their retrieved embeddings have an enriched covariance structure derived
from co-occurrences of features in the stored embeddings. However, modern
Hopfield networks increase the saturation effect of the InfoNCE objective which
hampers learning. We propose to use the InfoLOOB objective to mitigate this
saturation effect. We introduce the novel "Contrastive Leave One Out Boost"
(CLOOB), which uses modern Hopfield networks for covariance enrichment together
with the InfoLOOB objective. In experiments we compare CLOOB to CLIP after
pre-training on the Conceptual Captions and the YFCC dataset with respect to
their zero-shot transfer learning performance on other datasets. CLOOB
consistently outperforms CLIP at zero-shot transfer learning across all
considered architectures and datasets.

本研究提出使用 Hopfield 网络和 InfoLOOB 目标函数结合进行预训练来解决 CLIP 模型在零样本迁移学习中的过拟合问题，由实验得知，这种方法相比于仅使用 CLIP 模型，在多个架构和数据集上均具有更好的零样本迁移学习表现。