Contrastive self-supervised learning (CSL) with a prototypical regularization
has been introduced in learning meaningful representations for downstream tasks
that require strong semantic information. However, to optimize CSL with a loss
that performs the prototypical regularization aggressively, e.g., the ProtoNCE
loss, might cause the "coagulation" of examples in the embedding space. That
is, the intra-prototype diversity of samples collapses to trivial solutions for
their prototype being well-separated from others. Motivated by previous works,
we propose to mitigate this phenomenon by learning Prototypical representation
through Alignment, Uniformity and Correlation (PAUC). Specifically, the
ordinary ProtoNCE loss is revised with: (1) an alignment loss that pulls
embeddings from positive prototypes together; (2) a uniformity loss that
distributes the prototypical level features uniformly; (3) a correlation loss
that increases the diversity and discriminability between prototypical level
features. We conduct extensive experiments on various benchmarks where the
results demonstrate the effectiveness of our method in improving the quality of
prototypical contrastive representations. Particularly, in the classification
down-stream tasks with linear probes, our proposed method outperforms the
state-of-the-art instance-wise and prototypical contrastive learning methods on
the ImageNet-100 dataset by 2.96% and the ImageNet-1K dataset by 2.46% under
the same settings of batch size and epochs.

本文引入 PAUC 方法来学习基于原型的对比表示，以优化下游任务中需要强语义信息的有意义的表示。在各个基准测试中，实验结果证明了我们的方法在提高对比原型表示的质量方面的有效性，并在 ImageNet-100 数据集上将分类下游任务的性能提高了 2.96％，在 ImageNet-1K 数据集上提高了 2.46％。

通过对齐、均匀性和相关性重新思考原型对比学习

Rethinking Prototypical Contrastive Learning through Alignment, Uniformity and Correlation

Click-through rate (CTR) prediction plays an important role in online
advertising and recommender systems. In practice, the training of CTR models
depends on click data which is intrinsically biased towards higher positions
since higher position has higher CTR by nature. Existing methods such as actual
position training with fixed position inference and inverse propensity weighted
training with no position inference alleviate the bias problem to some extend.
However, the different treatment of position information between training and
inference will inevitably lead to inconsistency and sub-optimal online
performance. Meanwhile, the basic assumption of these methods, i.e., the click
probability is the product of examination probability and relevance
probability, is oversimplified and insufficient to model the rich interaction
between position and other information. In this paper, we propose a Deep
Position-wise Interaction Network (DPIN) to efficiently combine all candidate
items and positions for estimating CTR at each position, achieving consistency
between offline and online as well as modeling the deep non-linear interaction
among position, user, context and item under the limit of serving performance.
Following our new treatment to the position bias in CTR prediction, we propose
a new evaluation metrics named PAUC (position-wise AUC) that is suitable for
measuring the ranking quality at a given position. Through extensive
experiments on a real world dataset, we show empirically that our method is
both effective and efficient in solving position bias problem. We have also
deployed our method in production and observed statistically significant
improvement over a highly optimized baseline in a rigorous A/B test.

本文提出了一种深度位置交互网络 DPIN，以有效组合所有候选项和位置来估算每个位置的 CTR，实现离线和在线的一致性，并在服务性能的限制下对位置、用户、上下文和项目之间的深度非线性交互进行建模。同时，我们提出了一种评估指标 PAUC (position-wise AUC)，以解决 CTR 预测中的位置偏差问题，并在真实数据集上进行了广泛的实验，证明了方法的有效性和效率。在 A/B 测试中，我们的方法已在生产中部署，并观察到与高度优化的基准相比有显著的改进。