Backpropagation, while effective for gradient computation, falls short in
addressing memory consumption, limiting scalability. This work explores
forward-mode gradient computation as an alternative in invertible networks,
showing its potential to reduce the memory footprint without substantial
drawbacks. We introduce a novel technique based on a vector-inverse-Jacobian
product that accelerates the computation of forward gradients while retaining
the advantages of memory reduction and preserving the fidelity of true
gradients. Our method, Moonwalk, has a time complexity linear in the depth of
the network, unlike the quadratic time complexity of na\"ive forward, and
empirically reduces computation time by several orders of magnitude without
allocating more memory. We further accelerate Moonwalk by combining it with
reverse-mode differentiation to achieve time complexity comparable with
backpropagation while maintaining a much smaller memory footprint. Finally, we
showcase the robustness of our method across several architecture choices.
Moonwalk is the first forward-based method to compute true gradients in
invertible networks in computation time comparable to backpropagation and using
significantly less memory.

该研究论文探索了前向梯度计算作为可替代的反向传播方法，在可逆网络中减少内存占用的潜力，并引入了一种基于矢量 - 逆雅可比积的新技术，加速前向梯度计算并保持真实梯度的保真度。该方法在网络深度的时间复杂度是线性的，相比朴素前向传播的二次时间复杂度，计算时间可以显著降低，且不需要分配更多内存。通过将其与反向传播相结合，进一步加速了 Moonwalk，并在保持更小内存占用的同时达到与反向传播相当的时间复杂度。最后，论文展示了该方法在多种架构选择下的稳健性。Moonwalk 是第一个在可逆网络中计算真实梯度的前向方法，其计算时间与反向传播相当，并且占用的内存显著较少。