Sequence-to-sequence models usually transfer all encoder outputs to the
decoder for generation. In this work, by contrast, we hypothesize that these
encoder outputs can be compressed to shorten the sequence delivered for
decoding. We take Transformer as the testbed and introduce a layer of
stochastic gates in-between the encoder and the decoder. The gates are
regularized using the expected value of the sparsity-inducing L0penalty,
resulting in completely masking-out a subset of encoder outputs. In other
words, via joint training, the L0DROP layer forces Transformer to route
information through a subset of its encoder states. We investigate the effects
of this sparsification on two machine translation and two summarization tasks.
Experiments show that, depending on the task, around 40-70% of source encodings
can be pruned without significantly compromising quality. The decrease of the
output length endows L0DROP with the potential of improving decoding
efficiency, where it yields a speedup of up to 1.65x on document summarization
tasks against the standard Transformer. We analyze the L0DROP behaviour and
observe that it exhibits systematic preferences for pruning certain word types,
e.g., function words and punctuation get pruned most. Inspired by these
observations, we explore the feasibility of specifying rule-based patterns that
mask out encoder outputs based on information such as part-of-speech tags, word
frequency and word position.

利用 L0DROP 层来实现精简 Transformer 中 Encoder 输出的机制，可以在不降低结果质量的情况下减少 40-70% 的输出，提高编码速度。