Pretrained Masked Language Models (MLMs) have revolutionised NLP in recent
years. However, previous work has indicated that off-the-shelf MLMs are not
effective as universal lexical or sentence encoders without further
task-specific fine-tuning on NLI, sentence similarity, or paraphrasing tasks
using annotated task data. In this work, we demonstrate that it is possible to
turn MLMs into effective universal lexical and sentence encoders even without
any additional data and without any supervision. We propose an extremely
simple, fast and effective contrastive learning technique, termed Mirror-BERT,
which converts MLMs (e.g., BERT and RoBERTa) into such encoders in 20-30
seconds without any additional external knowledge. Mirror-BERT relies on fully
identical or slightly modified string pairs as positive (i.e., synonymous)
fine-tuning examples, and aims to maximise their similarity during identity
fine-tuning. We report huge gains over off-the-shelf MLMs with Mirror-BERT in
both lexical-level and sentence-level tasks, across different domains and
different languages. Notably, in the standard sentence semantic similarity
(STS) tasks, our self-supervised Mirror-BERT model even matches the performance
of the task-tuned Sentence-BERT models from prior work. Finally, we delve
deeper into the inner workings of MLMs, and suggest some evidence on why this
simple approach can yield effective universal lexical and sentence encoders.

本论文提出了一种名为 Mirror-BERT 的极为简单、快速和有效的对比学习技术，它能在 20-30 秒内将预训练的 MLMs（如 BERT 和 RoBERTa）转变为有效的通用词汇和句子编码器，而无需任何附加的数据和监督，并取得了巨大的成功。