Distributed data-parallel (DDP) training improves overall application
throughput as multiple devices train on a subset of data and aggregate updates
to produce a globally shared model. The periodic synchronization at each
iteration incurs considerable overhead, exacerbated by the increasing size and
complexity of state-of-the-art neural networks. Although many gradient
compression techniques propose to reduce communication cost, the ideal
compression factor that leads to maximum speedup or minimum data exchange
remains an open-ended problem since it varies with the quality of compression,
model size and structure, hardware, network topology and bandwidth. We propose
GraVAC, a framework to dynamically adjust compression factor throughout
training by evaluating model progress and assessing gradient information loss
associated with compression. GraVAC works in an online, black-box manner
without any prior assumptions about a model or its hyperparameters, while
achieving the same or better accuracy than dense SGD (i.e., no compression) in
the same number of iterations/epochs. As opposed to using a static compression
factor, GraVAC reduces end-to-end training time for ResNet101, VGG16 and LSTM
by 4.32x, 1.95x and 6.67x respectively. Compared to other adaptive schemes, our
framework provides 1.94x to 5.63x overall speedup.

本文介绍了 GraVAC，一种动态调整压缩因子的框架，用于在分布式数据并行训练中降低通信开销并提高训练速度。GraVAC 可以根据模型进展和梯度信息损失自适应地进行压缩，相较于静态压缩因子，可以将 ResNet101，VGG16 和 LSTM 的端到端训练时间分别缩短 4.32x，1.95x 和 6.67x，相较于其他自适应方案，整体加速比可达 1.94x 至 5.63x。

GraVAC：通信高效的分布式深度学习训练自适应压缩

GraVAC: Adaptive Compression for Communication-Efficient Distributed DL  Training

We present the Sequential Ensemble Transform (SET) method, an approach for
generating approximate samples from a Bayesian posterior distribution. The
method explores the posterior distribution by solving a sequence of discrete
optimal transport problems to produce a series of transport plans which map
prior samples to posterior samples. We prove that the sequence of Dirac mixture
distributions produced by the SET method converges weakly to the true posterior
as the sample size approaches infinity. Furthermore, our numerical results
indicate that, when compared to standard Sequential Monte Carlo (SMC) methods,
the SET approach is more robust to the choice of Markov mutation kernels and
requires less computational efforts to reach a similar accuracy when used to
explore complex posterior distributions. Finally, we describe adaptive schemes
that allow to completely automate the use of the SET method.

我们提出了顺序集成变换（SET）方法，这是一种从贝叶斯后验分布中生成近似样本的方法，该方法通过解决一系列离散最优传输问题，生成将先验样本映射到后验样本的一系列传输计划。我们证明了 SET 方法产生的 Dirac 混合分布序列在样本大小无限增大时弱收敛至真后验分布，并且数值结果表明，与标准的顺序蒙特卡罗（SMC）方法相比，SET 方法对于马尔科夫突变核的选择更加稳健，并且在探索复杂的后验分布时需要更少的计算量。最后，我们描述了自适应方案，可以完全自动化 SET 方法的使用。

贝叶斯反问题的序列集成变换

Sequential Ensemble Transform for Bayesian Inverse Problems

Several first order stochastic optimization methods commonly used in the
Euclidean domain such as stochastic gradient descent (SGD), accelerated
gradient descent or variance reduced methods have already been adapted to
certain Riemannian settings. However, some of the most popular of these
optimization tools - namely Adam , Adagrad and the more recent Amsgrad - remain
to be generalized to Riemannian manifolds. We discuss the difficulty of
generalizing such adaptive schemes to the most agnostic Riemannian setting, and
then provide algorithms and convergence proofs for geodesically convex
objectives in the particular case of a product of Riemannian manifolds, in
which adaptivity is implemented across manifolds in the cartesian product. Our
generalization is tight in the sense that choosing the Euclidean space as
Riemannian manifold yields the same algorithms and regret bounds as those that
were already known for the standard algorithms. Experimentally, we show faster
convergence and to a lower train loss value for Riemannian adaptive methods
over their corresponding baselines on the realistic task of embedding the
WordNet taxonomy in the Poincare ball.

将 Adam、Adagrad 和 Amsgrad 等流行的自适应随机优化方法扩展到里曼流形上面的困难以及基于里曼流形的优化算法和渐进结果的提出，同时在实验中证明该算法比原算法更快且表现更好。

黎曼自适应优化方法

Riemannian Adaptive Optimization Methods

An extension of the traditional two-armed bandit problem is considered, in
which the decision maker has access to some side information before deciding
which arm to pull. At each time t, before making a selection, the decision
maker is able to observe a random variable X_t that provides some information
on the rewards to be obtained. The focus is on finding uniformly good rules
(that minimize the growth rate of the inferior sampling time) and on
quantifying how much the additional information helps. Various settings are
considered and for each setting, lower bounds on the achievable inferior
sampling time are developed and asymptotically optimal adaptive schemes
achieving these lower bounds are constructed.

探讨传统的两臂赌博问题的扩展，其中决策者在选择拉动哪个手臂之前可以访问一些信息，在不同的设置中找到能够达到可接受水平的规则和构造渐近最优自适应方案以实现这些规则。