Feedforward computation, such as evaluating a neural network or sampling from
an autoregressive model, is ubiquitous in machine learning. The sequential
nature of feedforward computation, however, requires a strict order of
execution and cannot be easily accelerated with parallel computing. To enable
parallelization, we frame the task of feedforward computation as solving a
system of nonlinear equations. We then propose to find the solution using a
Jacobi or Gauss-Seidel fixed-point iteration method, as well as hybrid methods
of both. Crucially, Jacobi updates operate independently on each equation and
can be executed in parallel. Our method is guaranteed to give exactly the same
values as the original feedforward computation with a reduced (or equal) number
of parallelizable iterations, and hence reduced time given sufficient parallel
computing power. Experimentally, we demonstrate the effectiveness of our
approach in accelerating (i) backpropagation of RNNs, (ii) evaluation of
DenseNets, and (iii) autoregressive sampling of MADE and PixelCNN++, with
speedup factors between 2.1 and 26 under various settings.

本文章提出了使用 Jacobi 或 Gauss-Seidel 定点迭代方法来并行化前馈计算任务，以实现在神经网络评估和自回归模型采样等机器学习任务中的加速。实验表明，该方法在加速反向传播和评估 DenseNets 和 autoregressive sampling 任务中具有高效性，并在不同设置下显示出 2.1 至 26 倍的加速因子。

通过并行非线性方程求解加速前馈计算

Accelerating Feedforward Computation via Parallel Nonlinear Equation  Solving

We consider deep multi-layered generative models such as Boltzmann machines
or Hopfield nets in which computation (which implements inference) is both
recurrent and stochastic, but where the recurrence is not to model sequential
structure, only to perform computation. We find conditions under which a simple
feedforward computation is a very good initialization for inference, after the
input units are clamped to observed values. It means that after the feedforward
initialization, the recurrent network is very close to a fixed point of the
network dynamics, where the energy gradient is 0. The main condition is that
consecutive layers form a good auto-encoder, or more generally that different
groups of inputs into the unit (in particular, bottom-up inputs on one hand,
top-down inputs on the other hand) are consistent with each other, producing
the same contribution into the total weighted sum of inputs. In biological
terms, this would correspond to having each dendritic branch correctly
predicting the aggregate input from all the dendritic branches, i.e., the soma
potential. This is consistent with the prediction that the synaptic weights
into dendritic branches such as those of the apical and basal dendrites of
pyramidal cells are trained to minimize the prediction error made by the
dendritic branch when the target is the somatic activity. Whereas previous work
has shown how to achieve fast negative phase inference (when the model is
unclamped) in a predictive recurrent model, this contribution helps to achieve
fast positive phase inference (when the target output is clamped) in such
recurrent neural models.

探讨了深层次的生成模型，其中计算是既反复又随机的，但循环不是为了模拟顺序结构，而是为了执行计算，并提出了一种简单的前馈计算方法，可用于推断，使得循环神经网络非常接近网络动力学的固定点。