Pre-trained contextual language models are ubiquitously employed for language
understanding tasks, but are unsuitable for resource-constrained systems.
Noncontextual word embeddings are an efficient alternative in these settings.
Such methods typically use one vector to encode multiple different meanings of
a word, and incur errors due to polysemy. This paper proposes a two-stage
method to distill multiple word senses from a pre-trained language model (BERT)
by using attention over the senses of a word in a context and transferring this
sense information to fit multi-sense embeddings in a skip-gram-like framework.
We demonstrate an effective approach to training the sense disambiguation
mechanism in our model with a distribution over word senses extracted from the
output layer embeddings of BERT. Experiments on the contextual word similarity
and sense induction tasks show that this method is superior to or competitive
with state-of-the-art multi-sense embeddings on multiple benchmark data sets,
and experiments with an embedding-based topic model (ETM) demonstrates the
benefits of using this multi-sense embedding in a downstream application.

该研究提出了一种两阶段的方法，使用上下文中单词意义的注意力来提取一个单词的多重意义，并将其传递到 skip-gram 模型中，以生成多重意义的嵌入，这比现有技术表现更好，可应用于主题建模。

使用 BERT 知识蒸馏进行词义归纳

Word Sense Induction with Knowledge Distillation from BERT

We introduce Probabilistic FastText, a new model for word embeddings that can
capture multiple word senses, sub-word structure, and uncertainty information.
In particular, we represent each word with a Gaussian mixture density, where
the mean of a mixture component is given by the sum of n-grams. This
representation allows the model to share statistical strength across sub-word
structures (e.g. Latin roots), producing accurate representations of rare,
misspelt, or even unseen words. Moreover, each component of the mixture can
capture a different word sense. Probabilistic FastText outperforms both
FastText, which has no probabilistic model, and dictionary-level probabilistic
embeddings, which do not incorporate subword structures, on several
word-similarity benchmarks, including English RareWord and foreign language
datasets. We also achieve state-of-art performance on benchmarks that measure
ability to discern different meanings. Thus, the proposed model is the first to
achieve multi-sense representations while having enriched semantics on rare
words.

提出了一种新的单词嵌入模型 Probabilistic FastText，它可以同时捕捉到多个单词意义、子词结构和不确定性信息，使用高斯混合密度表示单词，包括词根和拼写错误等信息，比其他类似模型效果更好。