We propose a novel non-parametric/un-trainable language model, named
Non-Parametric Pairwise Attention Random Walk Model (NoPPA), to generate
sentence embedding only with pre-trained word embedding and pre-counted word
frequency. To the best we know, this study is the first successful attempt to
break the constraint on bag-of-words assumption with a non-parametric attention
mechanism. We evaluate our method on eight different downstream classification
tasks. The experiment results show that NoPPA outperforms all kinds of
bag-of-words-based methods in each dataset and provides a comparable or better
performance than the state-of-the-art non-parametric methods on average.
Furthermore, visualization supports that NoPPA can understand contextual
topics, common phrases, and word causalities. Our model is available at
this https URL.

提出一种名为 NoPPA 的新型非参数 / 不可训练语言模型，只使用预训练词嵌入和预计数词频来生成句子嵌入，并通过非参数的注意机制打破袋装词语假设的限制。在八个不同的下游分类任务上验证，实验结果显示 NoPPA 在每个数据集上均优于所有基于长袋的方法，并在平均性能上提供与最先进的非参数方法相当或更好的性能。此外，可视化证实了 NoPPA 能理解上下文主题、常用短语和词语因果关系。

无 PPA：非参数成对注意力随机游走模型的句子表示

NoPPA: Non-Parametric Pairwise Attention Random Walk Model for Sentence Representation

Topic modeling of textual corpora is an important and challenging problem. In
most previous work, the "bag-of-words" assumption is usually made which ignores
the ordering of words. This assumption simplifies the computation, but it
unrealistically loses the ordering information and the semantic of words in the
context. In this paper, we present a Gaussian Mixture Neural Topic Model
(GMNTM) which incorporates both the ordering of words and the semantic meaning
of sentences into topic modeling. Specifically, we represent each topic as a
cluster of multi-dimensional vectors and embed the corpus into a collection of
vectors generated by the Gaussian mixture model. Each word is affected not only
by its topic, but also by the embedding vector of its surrounding words and the
context. The Gaussian mixture components and the topic of documents, sentences
and words can be learnt jointly. Extensive experiments show that our model can
learn better topics and more accurate word distributions for each topic.
Quantitatively, comparing to state-of-the-art topic modeling approaches, GMNTM
obtains significantly better performance in terms of perplexity, retrieval
accuracy and classification accuracy.

本文提出了一种高斯混合神经主题模型（GMNTM），该模型将单词的顺序和句子的语义意义同时纳入了主题建模，实验结果表明，相比现有的主题建模方法，GMNTM 在困惑度、检索准确性和分类准确性方面获得了显著的改进。