Autoregressive decoding limits the efficiency of transformers for Machine
Translation (MT). The community proposed specific network architectures and
learning-based methods to solve this issue, which are expensive and require
changes to the MT model, trading inference speed at the cost of the translation
quality. In this paper, we propose to address the problem from the point of
view of decoding algorithms, as a less explored but rather compelling
direction. We propose to reframe the standard greedy autoregressive decoding of
MT with a parallel formulation leveraging Jacobi and Gauss-Seidel fixed-point
iteration methods for fast inference. This formulation allows to speed up
existing models without training or modifications while retaining translation
quality. We present three parallel decoding algorithms and test them on
different languages and models showing how the parallelization introduces a
speedup up to 38% w.r.t. the standard autoregressive decoding and nearly 2x
when scaling the method on parallel resources. Finally, we introduce a decoding
dependency graph visualizer (DDGviz) that let us see how the model has learned
the conditional dependence between tokens and inspect the decoding procedure.

本文提出使用雅可比和高斯 - 塞德尔迭代方法对机器翻译的标准贪心自回归解码进行并行重构，以解决 transformers 对机器翻译的效率问题，实现了速度提高，而翻译质量保持。三种并行解码算法被提出并在不同语言和模型上进行了测试，速度提升最多可达 38％，在并行资源上实现的速度几乎达到 2 倍。同时，为了更好地理解模型的条件依赖关系和检查解码过程，提出了解码依赖图可视化器（DDGviz）

通过并行解码加速 Transformer 推理以进行翻译

Accelerating Transformer Inference for Translation via Parallel Decoding

Feedforward computation, such as evaluating a neural network or sampling from
an autoregressive model, is ubiquitous in machine learning. The sequential
nature of feedforward computation, however, requires a strict order of
execution and cannot be easily accelerated with parallel computing. To enable
parallelization, we frame the task of feedforward computation as solving a
system of nonlinear equations. We then propose to find the solution using a
Jacobi or Gauss-Seidel fixed-point iteration method, as well as hybrid methods
of both. Crucially, Jacobi updates operate independently on each equation and
can be executed in parallel. Our method is guaranteed to give exactly the same
values as the original feedforward computation with a reduced (or equal) number
of parallelizable iterations, and hence reduced time given sufficient parallel
computing power. Experimentally, we demonstrate the effectiveness of our
approach in accelerating (i) backpropagation of RNNs, (ii) evaluation of
DenseNets, and (iii) autoregressive sampling of MADE and PixelCNN++, with
speedup factors between 2.1 and 26 under various settings.

本文章提出了使用 Jacobi 或 Gauss-Seidel 定点迭代方法来并行化前馈计算任务，以实现在神经网络评估和自回归模型采样等机器学习任务中的加速。实验表明，该方法在加速反向传播和评估 DenseNets 和 autoregressive sampling 任务中具有高效性，并在不同设置下显示出 2.1 至 26 倍的加速因子。

通过并行非线性方程求解加速前馈计算

Accelerating Feedforward Computation via Parallel Nonlinear Equation  Solving

We investigate the problem of optimal transport in the so-called Kantorovich
form, i.e. given two Radon measures on two compact sets, we seek an optimal
transport plan which is another Radon measure on the product of the sets that
has these two measures as marginals and minimizes a certain cost function. We
consider quadratic regularization of the problem, which forces the optimal
transport plan to be a square integrable function rather than a Radon measure.
We derive the dual problem and show strong duality and existence of primal and
dual solutions to the regularized problem. Then we derive two algorithms to
solve the dual problem of the regularized problem: A Gauss-Seidel method and a
semismooth quasi-Newton method and investigate both methods numerically. Our
experiments show that the methods perform well even for small regularization
parameters. Quadratic regularization is of interest since the resulting optimal
transport plans are sparse, i.e. they have a small support (which is not the
case for the often used entropic regularization where the optimal transport
plan always has full measure).

研究正则化问题的对偶问题，提出了用高斯 - 塞德尔方法和半光滑拟牛顿方法解决的方案，实验证明这两种方法在小正则化参数下表现良好。

二次正则化最优输运

Quadratically regularized optimal transport

The Kaczmarz and Gauss-Seidel methods both solve a linear system
$\bf{X}\bf{\beta} = \bf{y}$ by iteratively refining the solution estimate.
Recent interest in these methods has been sparked by a proof of Strohmer and
Vershynin which shows the randomized Kaczmarz method converges linearly in
expectation to the solution. Lewis and Leventhal then proved a similar result
for the randomized Gauss-Seidel algorithm. However, the behavior of both
methods depends heavily on whether the system is under or overdetermined, and
whether it is consistent or not. Here we provide a unified theory of both
methods, their variants for these different settings, and draw connections
between both approaches. In doing so, we also provide a proof that an extended
version of randomized Gauss-Seidel converges linearly to the least norm
solution in the underdetermined case (where the usual randomized Gauss Seidel
fails to converge). We detail analytically and empirically the convergence
properties of both methods and their extended variants in all possible system
settings. With this result, a complete and rigorous theory of both methods is
furnished.

本文提供了关于 Kaczmarz method 和 Gauss-Seidel method 的完整理论，证明了它们在不同情形下的收敛性和推导出其变种算法的收敛性，同时证明了 Gauss-Seidel 在欠定系统情形下收敛到最小范数解。