Despite its wide use, recent studies have revealed unexpected and undesirable
properties of neural autoregressive sequence models trained with maximum
likelihood, such as an unreasonably high affinity to short sequences after
training and to infinitely long sequences at decoding time. We propose to study
these phenomena by investigating how the modes, or local maxima, of a
distribution are maintained throughout the full learning chain of the
ground-truth, empirical, learned and decoding-induced distributions, via the
newly proposed mode recovery cost. We design a tractable testbed where we build
three types of ground-truth distributions: (1) an LSTM based structured
distribution, (2) an unstructured distribution where probability of a sequence
does not depend on its content, and (3) a product of these two which we call a
semi-structured distribution. Our study reveals both expected and unexpected
findings. First, starting with data collection, mode recovery cost strongly
relies on the ground-truth distribution and is most costly with the
semi-structured distribution. Second, after learning, mode recovery cost from
the ground-truth distribution may increase or decrease compared to data
collection, with the largest cost degradation occurring with the
semi-structured ground-truth distribution. Finally, the ability of the
decoding-induced distribution to recover modes from the learned distribution is
highly impacted by the choices made earlier in the learning chain. We conclude
that future research must consider the entire learning chain in order to fully
understand the potentials and perils and to further improve neural
autoregressive sequence models.

通过研究分布的模式如何在全学习链中维持来研究最大似然训练的神经自回归序列模型中的潜在问题，我们发现未来的研究必须考虑整个学习链，以充分了解神经自回归序列模型的潜力和危险，并进一步提高其性能。