Advances in the field of visual-language contrastive learning have made it
possible for many downstream applications to be carried out efficiently and
accurately by simply taking the dot product between image and text
representations. One of the most representative approaches proposed recently
known as CLIP has quickly garnered widespread adoption due to its
effectiveness. CLIP is trained with an InfoNCE loss that takes into account
both positive and negative samples to help learn a much more robust
representation space. This paper however reveals that the common downstream
practice of taking a dot product is only a zeroth-order approximation of the
optimization goal, resulting in a loss of information during test-time.
Intuitively, since the model has been optimized based on the InfoNCE loss,
test-time procedures should ideally also be in alignment. The question lies in
how one can retrieve any semblance of negative samples information during
inference. We propose Distribution Normalization (DN), where we approximate the
mean representation of a batch of test samples and use such a mean to represent
what would be analogous to negative samples in the InfoNCE loss. DN requires no
retraining or fine-tuning and can be effortlessly applied during inference.
Extensive experiments on a wide variety of downstream tasks exhibit a clear
advantage of DN over the dot product.

通常的下游做法只是优化目标的零阶近似，而 Distribution Normalization（DN）可以很好地检索到负样本的信息，从而在测试时间获得更好的性能。