We propose a novel non-parametric/un-trainable language model, named
Non-Parametric Pairwise Attention Random Walk Model (NoPPA), to generate
sentence embedding only with pre-trained word embedding and pre-counted word
frequency. To the best we know, this study is the first successful attempt to
break the constraint on bag-of-words assumption with a non-parametric attention
mechanism. We evaluate our method on eight different downstream classification
tasks. The experiment results show that NoPPA outperforms all kinds of
bag-of-words-based methods in each dataset and provides a comparable or better
performance than the state-of-the-art non-parametric methods on average.
Furthermore, visualization supports that NoPPA can understand contextual
topics, common phrases, and word causalities. Our model is available at
this https URL.

提出一种名为 NoPPA 的新型非参数 / 不可训练语言模型，只使用预训练词嵌入和预计数词频来生成句子嵌入，并通过非参数的注意机制打破袋装词语假设的限制。在八个不同的下游分类任务上验证，实验结果显示 NoPPA 在每个数据集上均优于所有基于长袋的方法，并在平均性能上提供与最先进的非参数方法相当或更好的性能。此外，可视化证实了 NoPPA 能理解上下文主题、常用短语和词语因果关系。

无 PPA：非参数成对注意力随机游走模型的句子表示

NoPPA: Non-Parametric Pairwise Attention Random Walk Model for Sentence Representation

Disambiguation of word senses in context is easy for humans, but is a major
challenge for automatic approaches. Sophisticated supervised and
knowledge-based models were developed to solve this task. However, (i) the
inherent Zipfian distribution of supervised training instances for a given word
and/or (ii) the quality of linguistic knowledge representations motivate the
development of completely unsupervised and knowledge-free approaches to word
sense disambiguation (WSD). They are particularly useful for under-resourced
languages which do not have any resources for building either supervised and/or
knowledge-based models. In this paper, we present a method that takes as input
a standard pre-trained word embedding model and induces a fully-fledged word
sense inventory, which can be used for disambiguation in context. We use this
method to induce a collection of sense inventories for 158 languages on the
basis of the original pre-trained fastText word embeddings by Grave et al.
(2018), enabling WSD in these languages. Models and system are available
online.

本研究提出了一种基于预训练的词嵌入，利用完全无监督和无基于知识的方法诱导一个完整的词义库，并实现对 158 种语言中的单词进行上下文消歧，对于资源匮乏的语言特别有用。