We introduce a method for embedding words as probability densities in a
low-dimensional space. Rather than assuming that a word embedding is fixed
across the entire text collection, as in standard word embedding methods, in
our Bayesian model we generate it from a word-specific prior density for each
occurrence of a given word. Intuitively, for each word, the prior density
encodes the distribution of its potential 'meanings'. These prior densities are
conceptually similar to Gaussian embeddings. Interestingly, unlike the Gaussian
embeddings, we can also obtain context-specific densities: they encode
uncertainty about the sense of a word given its context and correspond to
posterior distributions within our model. The context-dependent densities have
many potential applications: for example, we show that they can be directly
used in the lexical substitution task. We describe an effective estimation
method based on the variational autoencoding framework. We also demonstrate
that our embeddings achieve competitive results on standard benchmarks.

该研究介绍了一种将单词作为概率密度嵌入低维空间的方法，该 Bayesian 模型通过生成来自每个给定单词的单词特定先验密度，从而可以获得上下文特定的密度以及用于词汇替换任务的有效估算方法。

用贝叶斯 Skip-gram 模型将单词嵌入为分布

Embedding Words as Distributions with a Bayesian Skip-gram Model

Distributed word representations are widely used for modeling words in NLP
tasks. Most of the existing models generate one representation per word and do
not consider different meanings of a word. We present two approaches to learn
multiple topic-sensitive representations per word by using Hierarchical
Dirichlet Process. We observe that by modeling topics and integrating topic
distributions for each document we obtain representations that are able to
distinguish between different meanings of a given word. Our models yield
statistically significant improvements for the lexical substitution task
indicating that commonly used single word representations, even when combined
with contextual information, are insufficient for this task.

该论文提出使用分层狄利克雷过程学习每个单词的多个主题敏感表示的两种方法，通过模拟主题并集成每个文档的主题分布，可以得到能够区分给定单词不同含义的表示。该模型在词汇替换任务中表现出统计学显著的改进，表明常用的单词表示即使与上下文信息结合使用，也不足以完成该任务。