Probabilistic Circuits (PCs) are a unified framework for tractable
probabilistic models that support efficient computation of various
probabilistic queries (e.g., marginal probabilities). One key challenge is to
scale PCs to model large and high-dimensional real-world datasets: we observe
that as the number of parameters in PCs increases, their performance
immediately plateaus. This phenomenon suggests that the existing optimizers
fail to exploit the full expressive power of large PCs. We propose to overcome
such bottleneck by latent variable distillation: we leverage the less tractable
but more expressive deep generative models to provide extra supervision over
the latent variables of PCs. Specifically, we extract information from
Transformer-based generative models to assign values to latent variables of
PCs, providing guidance to PC optimizers. Experiments on both image and
language modeling benchmarks (e.g., ImageNet and WikiText-2) show that latent
variable distillation substantially boosts the performance of large PCs
compared to their counterparts without latent variable distillation. In
particular, on the image modeling benchmarks, PCs achieve competitive
performance against some of the widely-used deep generative models, including
variational autoencoders and flow-based models, opening up new avenues for
tractable generative modeling.

通过潜变量精炼机制在深度生成模型和 Probabilistic Circuits 之间建立联系，提高了 PCs 在大型和高维现实世界数据集上的性能表现，使其在图像建模中达到了与 VARIATIONAL AUTOENCODERS 和 FLOW-BASED MODELS 相当的竞争力，为可处理的生成建模开辟了新的途径。