Black Box Variational Inference is a promising framework in a succession of
recent efforts to make Variational Inference more ``black box". However, in
basic version it either fails to converge due to instability or requires some
fine-tuning of the update steps prior to execution that hinder it from being
completely general purpose. We propose a method for regulating its parameter
updates by reframing stochastic gradient ascent as a multivariate estimation
problem. We examine the properties of the James-Stein estimator as a
replacement for the arithmetic mean of Monte Carlo estimates of the gradient of
the evidence lower bound. The proposed method provides relatively weaker
variance reduction than Rao-Blackwellization, but offers a tradeoff of being
simpler and requiring no fine tuning on the part of the analyst. Performance on
benchmark datasets also demonstrate a consistent performance at par or better
than the Rao-Blackwellized approach in terms of model fit and time to
convergence.

黑盒变分推断是在使变分推断更 “黑盒” 的最近努力中的一个有前途的框架，但在基本版本中，它要么由于不稳定性而无法收敛，要么在执行前需要调整更新步骤，这使得它不完全通用。我们提出了一种通过将随机梯度上升重新定位为多元估计问题来规范其参数更新的方法。所提出的方法在方差减小方面相对较弱，但提供了更简单的代替和不需要分析师进行微调的权衡。基准数据集上的性能还表明，在模型拟合和收敛时间方面，与 Rao-Blackwell 化方法相比具有一致的表现或更好。

利用 James-Stein 估计器控制黑盒变分推断的方差

Variance Control for Black Box Variational Inference Using The  James-Stein Estimator

Stein's paradox holds considerable sway in high-dimensional statistics,
highlighting that the sample mean, traditionally considered the de facto
estimator, might not be the most efficacious in higher dimensions. To address
this, the James-Stein estimator proposes an enhancement by steering the sample
means toward a more centralized mean vector. In this paper, first, we establish
that normalization layers in deep learning use inadmissible estimators for mean
and variance. Next, we introduce a novel method to employ the James-Stein
estimator to improve the estimation of mean and variance within normalization
layers. We evaluate our method on different computer vision tasks: image
classification, semantic segmentation, and 3D object classification. Through
these evaluations, it is evident that our improved normalization layers
consistently yield superior accuracy across all tasks without extra
computational burden. Moreover, recognizing that a plethora of shrinkage
estimators surpass the traditional estimator in performance, we study two other
prominent shrinkage estimators: Ridge and LASSO. Additionally, we provide
visual representations to intuitively demonstrate the impact of shrinkage on
the estimated layer statistics. Finally, we study the effect of regularization
and batch size on our modified batch normalization. The studies show that our
method is less sensitive to batch size and regularization, improving accuracy
under various setups.

在本文中，我们首先确定深度学习中的归一化层使用的均值和方差估计器不可接受，然后介绍了一种运用 James-Stein 估计器改进均值和方差估计的新方法，评估表明我们改进的归一化层在各种计算机视觉任务中始终具有优越的准确性，而且不增加额外的计算负担。此外，我们还研究了两种显著的缩小估计器：Ridge 和 LASSO，通过可视化表示直观地展示了缩小对估计层统计量的影响，并对正则化和批量大小对我们修改后的批量归一化的影响进行了研究，研究显示我们的方法对批量大小和正则化不太敏感，在不同设置下提高了准确性。

使用 James-Stein 估计器改善归一化

Improving Normalization with the James-Stein Estimator

The James-Stein (JS) shrinkage estimator is a biased estimator that captures
the mean of Gaussian random vectors.While it has a desirable statistical
property of dominance over the maximum likelihood estimator (MLE) in terms of
mean squared error (MSE), not much progress has been made on extending the
estimator onto manifold-valued data.
We propose C-SURE, a novel Stein's unbiased risk estimate (SURE) of the JS
estimator on the manifold of complex-valued data with a theoretically proven
optimum over MLE. Adapting the architecture of the complex-valued SurReal
classifier, we further incorporate C-SURE into a prototype convolutional neural
network (CNN) classifier. We compare C-SURE with SurReal and a real-valued
baseline on complex-valued MSTAR and RadioML datasets.
C-SURE is more accurate and robust than SurReal, and the shrinkage estimator
is always better than MLE for the same prototype classifier. Like SurReal,
C-SURE is much smaller, outperforming the real-valued baseline on MSTAR
(RadioML) with less than 1 percent (3 percent) of the baseline size

我们提出了 C-SURE，这是一种新的 Stein 偏置风险估计器，用于复值数据的流型，并将其融入了卷积神经网络分类器的原型中，在复值 MSTAR 和 RadioML 数据集上表现比 SurReal 更准确和稳健