We provide theoretical and empirical evidence that using tighter evidence lower bounds (ELBOs) can be detrimental to the process of learning an inference network by reducing the signal-to-noise ratio of the gradient estimator. Our results call into question common implicit assumptions that tighter ELBOs are better variational objectives for simultaneous model learning and inference amortization schemes. Based on our insights, we introduce three new algorithms: the partially importance weighted auto-encoder (PIWAE), the multiply importance weighted auto-encoder (MIWAE), and the combination importance weighted auto-encoder (CIWAE), each of which includes the standard importance weighted auto-encoder (IWAE) as a special case. We show that each can deliver improvements over IWAE, even when performance is measured by the IWAE target itself. Moreover, PIWAE can simultaneously deliver improvements in both the quality of the inference network and generative network, relative to IWAE.

本文提供理论和实证证据表明，使用更严格的证据下界（ELBO）可能会降低梯度估计器的信噪比，从而对学习推理网络产生不利影响，并介绍了三种新算法：偏重要性加权自动编码器（PIWAE），乘法重要性加权自动编码器（MIWAE）和组合重要性加权自动编码器（CIWAE）。同时，我们的结果表明，PIWAE可以同时改善推理和生成网络的训练。

更紧的变分界并不一定更好