To simultaneously capture syntax and global semantics from a text corpus, we
propose a new larger-context recurrent neural network (RNN) based language
model, which extracts recurrent hierarchical semantic structure via a dynamic
deep topic model to guide natural language generation. Moving beyond a
conventional RNN-based language model that ignores long-range word dependencies
and sentence order, the proposed model captures not only intra-sentence word
dependencies, but also temporal transitions between sentences and
inter-sentence topic dependencies. For inference, we develop a hybrid of
stochastic-gradient Markov chain Monte Carlo and recurrent autoencoding
variational Bayes. Experimental results on a variety of real-world text corpora
demonstrate that the proposed model not only outperforms larger-context
RNN-based language models, but also learns interpretable recurrent multilayer
topics and generates diverse sentences and paragraphs that are syntactically
correct and semantically coherent.

提出了一种新的基于递归层次语义结构的流式话题模型的递归神经网络（RNN）语言模型，能够捕捉语料库中的语法和整体语义，并通过动态深度话题模型进行自然语言生成。实验结果表明，该模型不仅优于基于较大语境的 RNN 语言模型，而且能够学习可解释的递归多层话题，生成句法正确、语义连贯的多样句段。同时提出了一种融合随机梯度马尔科夫链蒙特卡洛和递归自编码变分贝叶斯的混合推理方法。

基于递归层次主题引导的循环神经网络语言生成模型

Recurrent Hierarchical Topic-Guided RNN for Language Generation

The Pachinko Allocation Machine (PAM) is a deep topic model that allows
representing rich correlation structures among topics by a directed acyclic
graph over topics. Because of the flexibility of the model, however,
approximate inference is very difficult. Perhaps for this reason, only a small
number of potential PAM architectures have been explored in the literature. In
this paper we present an efficient and flexible amortized variational inference
method for PAM, using a deep inference network to parameterize the approximate
posterior distribution in a manner similar to the variational autoencoder. Our
inference method produces more coherent topics than state-of-art inference
methods for PAM while being an order of magnitude faster, which allows
exploration of a wider range of PAM architectures than have previously been
studied.

本文介绍一种新方法 AM-PAM，使用基于深度变分自编码器的深度推断网络来实现 PAM 的高效灵活的摊销变分推理方法，并产生更连贯的主题。