Reversible architectures have been shown to be capable of performing on par
with their non-reversible architectures, being applied in deep learning for
memory savings and generative modeling. In this work, we show how reversible
architectures can solve challenges in parallelizing deep model training. We
introduce PETRA, a novel alternative to backpropagation for parallelizing
gradient computations. PETRA facilitates effective model parallelism by
enabling stages (i.e., a set of layers) to compute independently on different
devices, while only needing to communicate activations and gradients between
each other. By decoupling the forward and backward passes and keeping a single
updated version of the parameters, the need for weight stashing is also
removed. We develop a custom autograd-like training framework for PETRA, and we
demonstrate its effectiveness on CIFAR-10, ImageNet32, and ImageNet, achieving
competitive accuracies comparable to backpropagation using ResNet-18,
ResNet-34, and ResNet-50 models.

我们介绍了一种用于并行计算深度模型训练的 PETRA 替代方法，它通过反向传播和单一参数版本的保持来解决权重存储的问题，并在 CIFAR-10、ImageNet32 和 ImageNet 上展示了其与后向传播相比具有竞争性的准确性。

PETRA: 并行端到端训练与可逆架构

PETRA: Parallel End-to-end Training with Reversible Architectures

We propose PeTra, a memory-augmented neural network designed to track
entities in its memory slots. PeTra is trained using sparse annotation from the
GAP pronoun resolution dataset and outperforms a prior memory model on the task
while using a simpler architecture. We empirically compare key modeling
choices, finding that we can simplify several aspects of the design of the
memory module while retaining strong performance. To measure the people
tracking capability of memory models, we (a) propose a new diagnostic
evaluation based on counting the number of unique entities in text, and (b)
conduct a small scale human evaluation to compare evidence of people tracking
in the memory logs of PeTra relative to a previous approach. PeTra is highly
effective in both evaluations, demonstrating its ability to track people in its
memory despite being trained with limited annotation.

PeTra 是一款记忆增强的神经网络，旨在跟踪其记忆槽中的实体，并通过稀疏注释从 GAP 代词分辨率数据集中进行训练，其在任务上的表现优于先前的记忆模型。通过对关键建模选择进行实证比较，发现我们可以简化多个方面的内存模块设计同时保持强大性能。针对内存模型的人物追踪能力进行评估，提出了一种基于统计文本内唯一实体数量的新型诊断评估，并进行了小规模的人类评估，以比较 PeTra 与之前方法中内存日志中人物追踪方面的证据。PeTra 在两种评估中都非常有效，尽管受到有限的注释训练，但其能够在其内存中跟踪人物。