Self-supervised pre-training of language models usually consists in
predicting probability distributions over extensive token vocabularies. In this
study, we propose an innovative method that shifts away from probability
prediction and instead focuses on reconstructing input embeddings in a
contrastive fashion via Constrastive Weight Tying (CWT). We apply this approach
to pretrain Headless Language Models in both monolingual and multilingual
contexts. Our method offers practical advantages, substantially reducing
training computational requirements by up to 20 times, while simultaneously
enhancing downstream performance and data efficiency. We observe a significant
+1.6 GLUE score increase and a notable +2.7 LAMBADA accuracy improvement
compared to classical LMs within similar compute budgets.

自监督预训练语言模型通常通过对广泛的词汇表进行概率分布预测。在本研究中，我们提出了一种创新的方法，通过对比权重绑定（CWT）以对比的方式重构输入嵌入，从而摆脱了概率预测。我们将该方法应用于单语和多语境下预训练无头语言模型。我们的方法在减少训练计算需求高达 20 倍的同时，同时提高了下游性能和数据效率。与相似计算预算的传统语言模型相比，我们观察到显著的 + 1.6 GLUE 分数增加和显著的 + 2.7 LAMBADA 准确性提升。