We present a probabilistic language model for time-stamped text data which
tracks the semantic evolution of individual words over time. The model
represents words and contexts by latent trajectories in an embedding space. At
each moment in time, the embedding vectors are inferred from a probabilistic
version of word2vec [Mikolov et al., 2013]. These embedding vectors are
connected in time through a latent diffusion process. We describe two scalable
variational inference algorithms--skip-gram smoothing and skip-gram
filtering--that allow us to train the model jointly over all times; thus
learning on all data while simultaneously allowing word and context vectors to
drift. Experimental results on three different corpora demonstrate that our
dynamic model infers word embedding trajectories that are more interpretable
and lead to higher predictive likelihoods than competing methods that are based
on static models trained separately on time slices.

该研究提出了一种基于概率的语言模型，可以跟踪个别单词随时间的语义演变，并通过嵌入空间中的潜在轨迹表示单词和上下文。研究发现，该动态模型推断的单词嵌入轨迹更易解释，并且具有更高的预测性可能性。

动态词向量

Dynamic Word Embeddings

This paper presents a scalable method for integrating compositional
morphological representations into a vector-based probabilistic language model.
Our approach is evaluated in the context of log-bilinear language models,
rendered suitably efficient for implementation inside a machine translation
decoder by factoring the vocabulary. We perform both intrinsic and extrinsic
evaluations, presenting results on a range of languages which demonstrate that
our model learns morphological representations that both perform well on word
similarity tasks and lead to substantial reductions in perplexity. When used
for translation into morphologically rich languages with large vocabularies,
our models obtain improvements of up to 1.2 BLEU points relative to a baseline
system using back-off n-gram models.

本论文提出了一种可扩展的方法，将组成形态学表示集成到基于向量的概率语言模型中，并在语言模型因词汇而被适当地实现以实现内部和外部评估，在一些语言上进行实验研究并展示结果，表明我们的模型学习了形态学表示，在词相似性任务上表现出色且使困惑度大幅降低，用于转化大词汇量的形态丰富语言时，我们的模型相对于使用回退 N-gram 模型的基线系统，可以获得高达 1.2 的蓝点改进。