The application of neural network models to scientific machine learning tasks
has proliferated in recent years. In particular, neural network models have
proved to be adept at modeling processes with spatial-temporal complexity.
Nevertheless, these highly parameterized models have garnered skepticism in
their ability to produce outputs with quantified error bounds over the regimes
of interest. Hence there is a need to find uncertainty quantification methods
that are suitable for neural networks. In this work we present comparisons of
the parametric uncertainty quantification of neural networks modeling complex
spatial-temporal processes with Hamiltonian Monte Carlo and Stein variational
gradient descent and its projected variant. Specifically we apply these methods
to graph convolutional neural network models of evolving systems modeled with
recurrent neural network and neural ordinary differential equations
architectures. We show that Stein variational inference is a viable alternative
to Monte Carlo methods with some clear advantages for complex neural network
models. For our exemplars, Stein variational interference gave similar
uncertainty profiles through time compared to Hamiltonian Monte Carlo, albeit
with generally more generous variance.Projected Stein variational gradient
descent also produced similar uncertainty profiles to the non-projected
counterpart, but large reductions in the active weight space were confounded by
the stability of the neural network predictions and the convoluted likelihood
landscape.

神经网络模型在科学机器学习任务中的应用近年来已大量增加。尤其是，在建模具有时空复杂性的过程方面，神经网络模型表现出色。然而，这些高度参数化的模型在产生与感兴趣的区域保持定量化误差界限方面引起了怀疑。因此，有必要寻找适用于神经网络的不确定性量化方法。本研究对神经网络对复杂时空过程进行了参数不确定性量化的比较，采用了哈密顿蒙特卡洛和 Stein 变分梯度下降及其投影变体。具体而言，我们将这些方法应用于用递归神经网络和神经常微分方程模型建模的发展系统的图卷积神经网络模型。我们展示了 Stein 变分推断是复杂神经网络模型的可行替代方法。对于我们的示例，与哈密顿蒙特卡洛相比，Stein 变分推断在时间上给出了类似的不确定性描述，尽管方差普遍更大。投影 Stein 变分梯度下降也产生了与非投影对应的类似的不确定性描述，但是在神经网络预测的稳定性和复杂的似然函数空间中卷积产生了活动权重空间的大幅减少与困扰。

图卷积神经网络模型下演化过程的不确定性量化

Uncertainty Quantification of Graph Convolution Neural Network Models of  Evolving Processes

Network pruning is a promising way to generate light but accurate models and
enable their deployment on resource-limited edge devices. However, the current
state-of-the-art assumes that the effective sub-network and the other
superfluous parameters in the given network share the same distribution, where
pruning inevitably involves a distribution truncation operation. They usually
eliminate values near zero. While simple, it may not be the most appropriate
method, as effective models may naturally have many small values associated
with them. Removing near-zero values already embedded in model space may
significantly reduce model accuracy. Another line of work has proposed to
assign discrete prior over all possible sub-structures that still rely on
human-crafted prior hypotheses. Worse still, existing methods use regularized
point estimates, namely Hard Pruning, that can not provide error estimations
and fail reliability justification for the pruned networks. In this paper, we
propose a novel distribution-lossless pruning method, named DLLP, to
theoretically find the pruned lottery within Bayesian treatment. Specifically,
DLLP remodels the vanilla networks as discrete priors for the latent pruned
model and the other redundancy. More importantly, DLLP uses Stein Variational
Inference to approach the latent prior and effectively bypasses calculating KL
divergence with unknown distribution. Extensive experiments based on small
Cifar-10 and large-scaled ImageNet demonstrate that our method can obtain
sparser networks with great generalization performance while providing
quantified reliability for the pruned model.

本文提出了一种名为 DLLP 的新型网络剪枝方法，使用 Stein 变分推理和蒙特卡罗采样来避免对未知分布计算 KL 散度，该方法可以在不损失模型精度的情况下削减模型大小并提高通用性能。