Mixture variational distributions in black box variational inference (BBVI)
have demonstrated impressive results in challenging density estimation tasks.
However, currently scaling the number of mixture components can lead to a
linear increase in the number of learnable parameters and a quadratic increase
in inference time due to the evaluation of the evidence lower bound (ELBO). Our
two key contributions address these limitations. First, we introduce the novel
Multiple Importance Sampling Variational Autoencoder (MISVAE), which amortizes
the mapping from input to mixture-parameter space using one-hot encodings.
Fortunately, with MISVAE, each additional mixture component incurs a negligible
increase in network parameters. Second, we construct two new estimators of the
ELBO for mixtures in BBVI, enabling a tremendous reduction in inference time
with marginal or even improved impact on performance. Collectively, our
contributions enable scalability to hundreds of mixture components and provide
superior estimation performance in shorter time, with fewer network parameters
compared to previous Mixture VAEs. Experimenting with MISVAE, we achieve
astonishing, SOTA results on MNIST. Furthermore, we empirically validate our
estimators in other BBVI settings, including Bayesian phylogenetic inference,
where we improve inference times for the SOTA mixture model on eight data sets.

通过引入 Multiple Importance Sampling Variational Autoencoder（MISVAE）和构建两个新的 black box variational inference（BBVI）下的 ELBO 估计器，本文提出了解决了混合变分分布中元素个数扩展问题的两个方法，并在多个实验中验证了其可行性。

黑盒变分推断中的高效混合学习

Efficient Mixture Learning in Black-Box Variational Inference

Black Box Variational Inference is a promising framework in a succession of
recent efforts to make Variational Inference more ``black box". However, in
basic version it either fails to converge due to instability or requires some
fine-tuning of the update steps prior to execution that hinder it from being
completely general purpose. We propose a method for regulating its parameter
updates by reframing stochastic gradient ascent as a multivariate estimation
problem. We examine the properties of the James-Stein estimator as a
replacement for the arithmetic mean of Monte Carlo estimates of the gradient of
the evidence lower bound. The proposed method provides relatively weaker
variance reduction than Rao-Blackwellization, but offers a tradeoff of being
simpler and requiring no fine tuning on the part of the analyst. Performance on
benchmark datasets also demonstrate a consistent performance at par or better
than the Rao-Blackwellized approach in terms of model fit and time to
convergence.

黑盒变分推断是在使变分推断更 “黑盒” 的最近努力中的一个有前途的框架，但在基本版本中，它要么由于不稳定性而无法收敛，要么在执行前需要调整更新步骤，这使得它不完全通用。我们提出了一种通过将随机梯度上升重新定位为多元估计问题来规范其参数更新的方法。所提出的方法在方差减小方面相对较弱，但提供了更简单的代替和不需要分析师进行微调的权衡。基准数据集上的性能还表明，在模型拟合和收敛时间方面，与 Rao-Blackwell 化方法相比具有一致的表现或更好。

利用 James-Stein 估计器控制黑盒变分推断的方差

Variance Control for Black Box Variational Inference Using The  James-Stein Estimator

We introduce Kalman Gradient Descent, a stochastic optimization algorithm
that uses Kalman filtering to adaptively reduce gradient variance in stochastic
gradient descent by filtering the gradient estimates. We present both a
theoretical analysis of convergence in a non-convex setting and experimental
results which demonstrate improved performance on a variety of machine learning
areas including neural networks and black box variational inference. We also
present a distributed version of our algorithm that enables large-dimensional
optimization, and we extend our algorithm to SGD with momentum and RMSProp.

介绍一种使用卡尔曼过滤器进行随机优化的算法，并分析了其在非凸设置下收敛性的理论，并在神经网络和黑盒变分推理等许多机器学习领域上展示了其改进的性能。同时，介绍了一种分布式版本的算法，并将其扩展到 SGD 动量和 RMSProp。

Kalman 梯度下降：随机优化中自适应方差减小

Kalman Gradient Descent: Adaptive Variance Reduction in Stochastic  Optimization

Black box variational inference (BBVI) with reparameterization gradients
triggered the exploration of divergence measures other than the
Kullback-Leibler (KL) divergence, such as alpha divergences. In this paper, we
view BBVI with generalized divergences as a form of estimating the marginal
likelihood via biased importance sampling. The choice of divergence determines
a bias-variance trade-off between the tightness of a bound on the marginal
likelihood (low bias) and the variance of its gradient estimators. Drawing on
variational perturbation theory of statistical physics, we use these insights
to construct a family of new variational bounds. Enumerated by an odd integer
order $K$, this family captures the standard KL bound for $K=1$, and converges
to the exact marginal likelihood as $K\to\infty$. Compared to
alpha-divergences, our reparameterization gradients have a lower variance. We
show in experiments on Gaussian Processes and Variational Autoencoders that the
new bounds are more mass covering, and that the resulting posterior covariances
are closer to the true posterior and lead to higher likelihoods on held-out
data.

本文针对黑盒变分推断方法提出了基于广义差异度量的偏置重要性采样方法，并通过变分扰动理论构建了一族新的变分上界，对多个模型进行实验得出新的变分上界在数据拟合上更优