The ability to learn from large unlabeled corpora has allowed neural language
models to advance the frontier in natural language understanding. However,
existing self-supervision techniques operate at the word form level, which
serves as a surrogate for the underlying semantic content. This paper proposes
a method to employ weak-supervision directly at the word sense level. Our
model, named SenseBERT, is pre-trained to predict not only the masked words but
also their WordNet supersenses. Accordingly, we attain a lexical-semantic level
language model, without the use of human annotation. SenseBERT achieves
significantly improved lexical understanding, as we demonstrate by
experimenting on SemEval Word Sense Disambiguation, and by attaining a state of
the art result on the Word in Context task.

本文提出了一种名为 SenseBERT 的神经语言模型，通过利用单词意义级别直接进行弱监督，预测掩码单词及其 WordNet 超级意义，达到了词汇语义级别的语言模型，实验结果显示，在 SemEval Word Sense Disambiguation 任务和单词上下文任务中，SenseBERT 取得了显著的改进和最新的成果。