Self-supervised pretraining on unlabeled data followed by supervised
finetuning on labeled data is a popular paradigm for learning from limited
labeled examples. In this paper, we investigate and extend this paradigm to the
classical positive unlabeled (PU) setting - the weakly supervised task of
learning a binary classifier only using a few labeled positive examples and a
set of unlabeled samples. We propose a novel PU learning objective positive
unlabeled Noise Contrastive Estimation (puNCE) that leverages the available
explicit (from labeled samples) and implicit (from unlabeled samples)
supervision to learn useful representations from positive unlabeled input data.
The underlying idea is to assign each training sample an individual weight;
labeled positives are given unit weight; unlabeled samples are duplicated, one
copy is labeled positive and the other as negative with weights $\pi$ and
$(1-\pi)$ where $\pi$ denotes the class prior. Extensive experiments across
vision and natural language tasks reveal that puNCE consistently improves over
existing unsupervised and supervised contrastive baselines under limited
supervision.

这篇论文研究并扩展了自监督预训练在受限标注数据上的应用，提出了一种新颖的 PU 学习目标正无标噪声对比估计（puNCE）以通过有限监督获得表示，实验证明 puNCE 在受限监督下持续优于现有无监督和监督对比基线。