Noise Contrastive Estimation (NCE) is a powerful parameter estimation method for log-linear models, which avoids calculation of the partition function or its derivatives at each training step, a computationally demanding step in many cases. It is closely related to negative sampling methods, now widely used in NLP. This paper considers NCE-based estimation of conditional models. Conditional models are frequently encountered in practice; however there has not been a rigorous theoretical analysis of NCE in this setting, and we will argue there are subtle but important questions when generalizing NCE to the conditional case. In particular, we analyze two variants of NCE for conditional models: one based on a classification objective, the other based on a ranking objective. We show that the ranking-based variant of NCE gives consistent parameter estimates under weaker assumptions than the classification-based method; we analyze the statistical efficiency of the ranking-based and classification-based variants of NCE; finally we describe experiments on synthetic data and language modeling showing the effectiveness and trade-offs of both methods.

本文考虑了基于NCE的条件模型估计，通过分析两种条件模型变体：一种基于分类目标，另一种基于排序目标，证明了基于排序的NCE变体在较弱的假设下提供了一致的参数估计，同时分析了两种方法的统计效率，并在合成数据和语言建模实验中展示了两种方法的效果和权衡。

条件模型的噪声对比估计和负采样：一致性和统计效率