Automatic differentiation (AD) in reverse mode (RAD) is a central component
of deep learning and other uses of large-scale optimization. Commonly used RAD
algorithms such as backpropagation, however, are complex and stateful,
hindering deep understanding, improvement, and parallel execution. This paper
develops a simple, generalized AD algorithm calculated from a simple, natural
specification. The general algorithm is then specialized by varying the
representation of derivatives. In particular, applying well-known constructions
to a naive representation yields two RAD algorithms that are far simpler than
previously known. In contrast to commonly used RAD implementations, the
algorithms defined here involve no graphs, tapes, variables, partial
derivatives, or mutation. They are inherently parallel-friendly, correct by
construction, and usable directly from an existing programming language with no
need for new data types or programming style, thanks to use of an AD-agnostic
compiler plugin.

本文提出了一种由简单、自然的规范计算出的通用 AD 算法，通过使用 AD - 不可知编译器插件，这些算法可以直接从现有的编程语言中使用，无需新的数据类型或编程风格，并且不涉及任何图表、磁带、变量、偏导数或突变，从而更易于理解、改进和并行执行。

自动微分的简单本质

The simple essence of automatic differentiation

Automatic differentiation---the mechanical transformation of numeric computer
programs to calculate derivatives efficiently and accurately---dates to the
origin of the computer age. Reverse mode automatic differentiation both
antedates and generalizes the method of backwards propagation of errors used in
machine learning. Despite this, practitioners in a variety of fields, including
machine learning, have been little influenced by automatic differentiation, and
make scant use of available tools. Here we review the technique of automatic
differentiation, describe its two main modes, and explain how it can benefit
machine learning practitioners. To reach the widest possible audience our
treatment assumes only elementary differential calculus, and does not assume
any knowledge of linear algebra.

本文介绍了自动微分技术及其在机器学习中的应用，包括其两种主要模式和能够提高计算效率的优势。该技术具有普遍适用性，不需要先验知识，值得广泛的应用。