Learning unnormalized statistical models (e.g., energy-based models) is
computationally challenging due to the complexity of handling the partition
function. To eschew this complexity, noise-contrastive estimation~(NCE) has
been proposed by formulating the objective as the logistic loss of the real
data and the artificial noise. However, as found in previous works, NCE may
perform poorly in many tasks due to its flat loss landscape and slow
convergence. In this paper, we study it a direct approach for optimizing the
negative log-likelihood of unnormalized models from the perspective of
compositional optimization. To tackle the partition function, a noise
distribution is introduced such that the log partition function can be written
as a compositional function whose inner function can be estimated with
stochastic samples. Hence, the objective can be optimized by stochastic
compositional optimization algorithms. Despite being a simple method, we
demonstrate that it is more favorable than NCE by (1) establishing a fast
convergence rate and quantifying its dependence on the noise distribution
through the variance of stochastic estimators; (2) developing better results
for one-dimensional Gaussian mean estimation by showing our objective has a
much favorable loss landscape and hence our method enjoys faster convergence;
(3) demonstrating better performance on multiple applications, including
density estimation, out-of-distribution detection, and real image generation.

本文探讨了一种直接优化非规范化模型负对数似然的方法，通过引入噪声分布，实现了将对数分区函数写为组合函数的形式，进而通过随机组合优化算法进行优化，较噪声对比估计方法具有更快的收敛速度和更好的性能。