This paper studies the qualitative behavior and robustness of two variants of
Minimal Random Code Learning (MIRACLE) used to compress variational Bayesian
neural networks. MIRACLE implements a powerful, conditionally Gaussian
variational approximation for the weight posterior $Q_{\mathbf{w}}$ and uses
relative entropy coding to compress a weight sample from the posterior using a
Gaussian coding distribution $P_{\mathbf{w}}$. To achieve the desired
compression rate, $D_{\mathrm{KL}}[Q_{\mathbf{w}} \Vert P_{\mathbf{w}}]$ must
be constrained, which requires a computationally expensive annealing procedure
under the conventional mean-variance (Mean-Var) parameterization for
$Q_{\mathbf{w}}$. Instead, we parameterize $Q_{\mathbf{w}}$ by its mean and KL
divergence from $P_{\mathbf{w}}$ to constrain the compression cost to the
desired value by construction. We demonstrate that variational training with
Mean-KL parameterization converges twice as fast and maintains predictive
performance after compression. Furthermore, we show that Mean-KL leads to more
meaningful variational distributions with heavier tails and compressed weight
samples which are more robust to pruning.

研究了 MIRACLE 用于压缩变分贝叶斯神经网络的两个变体的定性行为和稳健性。使用均值 - KL 参数化，可以实现精准地控制压缩成本，并且训练更快、预测性能更好、具有更重的尾部和更稳健的压缩权重样本的含义更明确的变分分布，并进一步证明了这种方式具有更好的稳健性。