Pre-trained contextual language models are ubiquitously employed for language
understanding tasks, but are unsuitable for resource-constrained systems.
Noncontextual word embeddings are an efficient alternative in these settings.
Such methods typically use one vector to encode multiple different meanings of
a word, and incur errors due to polysemy. This paper proposes a two-stage
method to distill multiple word senses from a pre-trained language model (BERT)
by using attention over the senses of a word in a context and transferring this
sense information to fit multi-sense embeddings in a skip-gram-like framework.
We demonstrate an effective approach to training the sense disambiguation
mechanism in our model with a distribution over word senses extracted from the
output layer embeddings of BERT. Experiments on the contextual word similarity
and sense induction tasks show that this method is superior to or competitive
with state-of-the-art multi-sense embeddings on multiple benchmark data sets,
and experiments with an embedding-based topic model (ETM) demonstrates the
benefits of using this multi-sense embedding in a downstream application.

该研究提出了一种两阶段的方法，使用上下文中单词意义的注意力来提取一个单词的多重意义，并将其传递到 skip-gram 模型中，以生成多重意义的嵌入，这比现有技术表现更好，可应用于主题建模。