Deep learning has seen tremendous success over the past decade in computer
vision, machine translation, and gameplay. This success rests in crucial ways
on gradient-descent optimization and the ability to learn parameters of a
neural network by backpropagating observed errors. However, neural network
architectures are growing increasingly sophisticated and diverse, which
motivates an emerging quest for even more general forms of differentiable
programming, where arbitrary parameterized computations can be trained by
gradient descent. In this paper, we take a fresh look at automatic
differentiation (AD) techniques, and especially aim to demystify the
reverse-mode form of AD that generalizes backpropagation in neural networks.
We uncover a tight connection between reverse-mode AD and delimited
continuations, which permits implementing reverse-mode AD purely via operator
overloading and without any auxiliary data structures. We further show how this
formulation of AD can be fruitfully combined with multi-stage programming
(staging), leading to a highly efficient implementation that combines the
performance benefits of deep learning frameworks based on explicit reified
computation graphs (e.g., TensorFlow) with the expressiveness of pure library
approaches (e.g., PyTorch).

本文研究了自动微分技术和其与限定续延之间的紧密关系，提出了无需任何辅助数据结构的运算符重载形式的反向模式自动微分实现，并将其与多阶段编程相结合，实现了 TensorFlow 等框架的计算性能优势和 PyTorch 等库的表达能力。