Extensive research has recently shown that recurrent neural language models
are able to process a wide range of grammatical phenomena. How these models are
able to perform these remarkable feats so well, however, is still an open
question. To gain more insight into what information LSTMs base their decisions
on, we propose a generalisation of Contextual Decomposition (GCD). In
particular, this setup enables us to accurately distil which part of a
prediction stems from semantic heuristics, which part truly emanates from
syntactic cues and which part arise from the model biases themselves instead.
We investigate this technique on tasks pertaining to syntactic agreement and
co-reference resolution and discover that the model strongly relies on a
default reasoning effect to perform these tasks.

本文研究了递归神经语言模型及其在语法现象处理方面的应用，并提出了基于上下文分解的泛化方法来更好地分析模型的决策基础，发现模型在处理语法一致性及指代消解任务时往往依赖于默认推理效应。

分析神经语言模型：上下文分解揭示数字和性别指派中的默认推理

Analysing Neural Language Models: Contextual Decomposition Reveals  Default Reasoning in Number and Gender Assignment

Existing recurrent neural language models often fail to capture higher-level
structure present in text: for example, rhyming patterns present in poetry.
Much prior work on poetry generation uses manually defined constraints which
are satisfied during decoding using either specialized decoding procedures or
rejection sampling. The rhyming constraints themselves are typically not
learned by the generator. We propose an alternate approach that uses a
structured discriminator to learn a poetry generator that directly captures
rhyming constraints in a generative adversarial setup. By causing the
discriminator to compare poems based only on a learned similarity matrix of
pairs of line ending words, the proposed approach is able to successfully learn
rhyming patterns in two different English poetry datasets (Sonnet and Limerick)
without explicitly being provided with any phonetic information.

使用结构化判别器的生成式对抗网络实现押韵约束的诗歌生成模型。

使用结构对抗学习押韵约束

Learning Rhyming Constraints using Structured Adversaries

Recurrent neural language models are the state-of-the-art models for language
modeling. When the vocabulary size is large, the space taken to store the model
parameters becomes the bottleneck for the use of recurrent neural language
models. In this paper, we introduce a simple space compression method that
randomly shares the structured parameters at both the input and output
embedding layers of the recurrent neural language models to significantly
reduce the size of model parameters, but still compactly represent the original
input and output embedding layers. The method is easy to implement and tune.
Experiments on several data sets show that the new method can get similar
perplexity and BLEU score results while only using a very tiny fraction of
parameters.

本文介绍了一种通过在 RNN 中的输入和输出嵌入层共享参数来压缩模型参数的方法，实验证明该方法可大幅减小模型参数的大小，但仍能在自然语言处理中保持模型的准确性和性能表现。