Recurrent Neural Networks (RNNs) with attention mechanisms have obtained
state-of-the-art results for many sequence processing tasks. Most of these
models use a simple form of encoder with attention that looks over the entire
sequence and assigns a weight to each token independently. We present a
mechanism for focusing RNN encoders for sequence modelling tasks which allows
them to attend to key parts of the input as needed. We formulate this using a
multi-layer conditional sequence encoder that reads in one token at a time and
makes a discrete decision on whether the token is relevant to the context or
question being asked. The discrete gating mechanism takes in the context
embedding and the current hidden state as inputs and controls information flow
into the layer above. We train it using policy gradient methods. We evaluate
this method on several types of tasks with different attributes. First, we
evaluate the method on synthetic tasks which allow us to evaluate the model for
its generalization ability and probe the behavior of the gates in more
controlled settings. We then evaluate this approach on large scale Question
Answering tasks including the challenging MS MARCO and SearchQA tasks. Our
models shows consistent improvements for both tasks over prior work and our
baselines. It has also shown to generalize significantly better on synthetic
tasks as compared to the baselines.

提出了使用基于注意力机制的人工神经网络（RNNs）来实现序列建模任务的机制，它允许 RNNs 关注需要的输入的关键部分。通过使用一个多层的有条件序列编码器，读取一个令牌并对其进行离散的决策，来控制信息流入以上层。此方法在几种任务类型上进行评估，包括各种属性的合成任务以及大规模的问答任务，并且相对于已有的基准模型，都得到了一致的性能改进。