Globally normalized neural sequence models are considered superior to their
locally normalized equivalents because they may ameliorate the effects of label
bias. However, when considering high-capacity neural parametrizations that
condition on the whole input sequence, both model classes are theoretically
equivalent in terms of the distributions they are capable of representing.
Thus, the practical advantage of global normalization in the context of modern
neural methods remains unclear. In this paper, we attempt to shed light on this
problem through an empirical study. We extend an approach for search-aware
training via a continuous relaxation of beam search (Goyal et al., 2017b) in
order to enable training of globally normalized recurrent sequence models
through simple backpropagation. We then use this technique to conduct an
empirical study of the interaction between global normalization, high-capacity
encoders, and search-aware optimization. We observe that in the context of
inexact search, globally normalized neural models are still more effective than
their locally normalized counterparts. Further, since our training approach is
sensitive to warm-starting with pre-trained models, we also propose a novel
initialization strategy based on self-normalization for pre-training globally
normalized models. We perform analysis of our approach on two tasks: CCG
supertagging and Machine Translation, and demonstrate the importance of global
normalization under different conditions while using search-aware training.

本研究通过对搜索感知训练的扩展，实现对全局标准化经常序列模型的训练，并在 CCG supertagging 和机器翻译任务中证明了全局标准化在不同条件下的重要性。