Variational inference (VI) can be cast as an optimization problem in which
the variational parameters are tuned to closely align a variational
distribution with the true posterior. The optimization task can be approached
through vanilla gradient descent in black-box VI or natural-gradient descent in
natural-gradient VI. In this work, we reframe VI as the optimization of an
objective that concerns probability distributions defined over a
\textit{variational parameter space}. Subsequently, we propose Wasserstein
gradient descent for tackling this optimization problem. Notably, the
optimization techniques, namely black-box VI and natural-gradient VI, can be
reinterpreted as specific instances of the proposed Wasserstein gradient
descent. To enhance the efficiency of optimization, we develop practical
methods for numerically solving the discrete gradient flows. We validate the
effectiveness of the proposed methods through empirical experiments on a
synthetic dataset, supplemented by theoretical analyses.

Variational inference (变分推断) can be optimized using Wasserstein gradient descent methods to improve efficiency and alignment of variational parameters with the true posterior.

变分推断中的变分参数空间上的 Wasserstein 梯度流

Wasserstein Gradient Flow over Variational Parameter Space for  Variational Inference

Neural network (NN) designed for challenging machine learning tasks is in
general a highly nonlinear mapping that contains massive variational
parameters. High complexity of NN, if unbounded or unconstrained, might
unpredictably cause severe issues including over-fitting, loss of
generalization power, and unbearable cost of hardware. In this work, we propose
a general compression scheme that significantly reduces the variational
parameters of NN by encoding them to multi-layer tensor networks (TN's) that
contain exponentially-fewer free parameters. Superior compression performance
of our scheme is demonstrated on several widely-recognized NN's (FC-2, LeNet-5,
and VGG-16) and datasets (MNIST and CIFAR-10), surpassing the state-of-the-art
method based on shallow tensor networks. For instance, about 10 million
parameters in the three convolutional layers of VGG-16 are compressed in TN's
with just $632$ parameters, while the testing accuracy on CIFAR-10 is
surprisingly improved from $81.14\%$ by the original NN to $84.36\%$ after
compression. Our work suggests TN as an exceptionally efficient mathematical
structure for representing the variational parameters of NN's, which superiorly
exploits the compressibility than the simple multi-way arrays.

本文提出了一种基于张量网络的压缩算法，能够显著地降低神经网络的参数量，从而提高其压缩效果和泛化性能。实验证明，该算法可将包含上千万参数的 VGG-16 模型的卷积层压缩为仅包含 632 个参数的张量网络，同时提升在 CIFAR-10 数据集上的测试准确率。该算法可被看作是一种高效的神经网络参数压缩方案，能够充分挖掘神经网络的可压缩性。

通过张量网络将神经网络压缩为指数少的变分参数

Compressing neural network by tensor network with exponentially fewer  variational parameters

We propose a method (TT-GP) for approximate inference in Gaussian Process
(GP) models. We build on previous scalable GP research including stochastic
variational inference based on inducing inputs, kernel interpolation, and
structure exploiting algebra. The key idea of our method is to use Tensor Train
decomposition for variational parameters, which allows us to train GPs with
billions of inducing inputs and achieve state-of-the-art results on several
benchmarks. Further, our approach allows for training kernels based on deep
neural networks without any modifications to the underlying GP model. A neural
network learns a multidimensional embedding for the data, which is used by the
GP to make the final prediction. We train GP and neural network parameters
end-to-end without pretraining, through maximization of GP marginal likelihood.
We show the efficiency of the proposed approach on several regression and
classification benchmark datasets including MNIST, CIFAR-10, and Airline.

利用张量列车分解为变分参数的高斯过程建模方法，实现具有数十亿个输入点培训和端到端训练的深度神经网络和高斯过程回归 / 分类器的最新结果优化。