Self-normalizing discriminative models approximate the normalized probability
of a class without having to compute the partition function. In the context of
language modeling, this property is particularly appealing as it may
significantly reduce run-times due to large word vocabularies. In this study,
we provide a comprehensive investigation of language modeling
self-normalization. First, we theoretically analyze the inherent
self-normalization properties of Noise Contrastive Estimation (NCE) language
models. Then, we compare them empirically to softmax-based approaches, which
are self-normalized using explicit regularization, and suggest a hybrid model
with compelling properties. Finally, we uncover a surprising negative
correlation between self-normalization and perplexity across the board, as well
as some regularity in the observed errors, which may potentially be used for
improving self-normalization algorithms in the future.

本研究对语言建模中的自归一化进行了全面的研究，通过理论分析噪声对比估计 (NCE) 语言模型的自归一化特性，与使用显式正则化来自归一化的 softmax-based 方法进行了经验比较，并提出了一种具有吸引力的混合模型。最后，发现自归一化和困惑度之间存在负相关，这一规律为未来改善自归一化算法提供了一定的帮助。