Recent efforts in applying implicit networks to solve inverse problems in
imaging have achieved competitive or even superior results when compared to
feedforward networks. These implicit networks only require constant memory
during backpropagation, regardless of the number of layers. However, they are
not necessarily easy to train. Gradient calculations are computationally
expensive because they require backpropagating through a fixed point. In
particular, this process requires solving a large linear system whose size is
determined by the number of features in the fixed point iteration. This paper
explores a recently proposed method, Jacobian-free Backpropagation (JFB), a
backpropagation scheme that circumvents such calculation, in the context of
image deblurring problems. Our results show that JFB is comparable against
fine-tuned optimization schemes, state-of-the-art (SOTA) feedforward networks,
and existing implicit networks at a reduced computational cost.

通过在图像去模糊问题的背景下探索最近提出的无雅可比反向传播（Jacobian-free Backpropagation，JFB）方法，本文展示了 JFB 在减少计算成本方面与优化方案、最先进的前馈网络以及现有的隐式网络具有可比性。

使用无雅各比传递正向传播训练隐式网络进行图像去模糊

Training Implicit Networks for Image Deblurring using Jacobian-Free  Backpropagation

In appropriate frameworks, automatic differentiation is transparent to the
user at the cost of being a significant computational burden when the number of
operations is large. For iterative algorithms, implicit differentiation
alleviates this issue but requires custom implementation of Jacobian
evaluation. In this paper, we study one-step differentiation, also known as
Jacobian-free backpropagation, a method as easy as automatic differentiation
and as performant as implicit differentiation for fast algorithms (e.g.,
superlinear optimization methods). We provide a complete theoretical
approximation analysis with specific examples (Newton's method, gradient
descent) along with its consequences in bilevel optimization. Several numerical
examples illustrate the well-foundness of the one-step estimator.

本文提出一种新的自动求导方法 —— 一步法微分（Jacobian-free backpropagation），其性能可与隐式微分方法相媲美，并为快速算法（如超线性优化方法）提供了解决方案。其中使用特定的例子（如牛顿法和梯度下降法）对其进行全面的理论近似分析，并揭示了其在双层优化中的应用。通过多个数值示例，证明了这种一步估计器的正确性。

迭代算法的一步差分

One-step differentiation of iterative algorithms

A promising trend in deep learning replaces traditional feedforward networks
with implicit networks. Unlike traditional networks, implicit networks solve a
fixed point equation to compute inferences. Solving for the fixed point varies
in complexity, depending on provided data and an error tolerance. Importantly,
implicit networks may be trained with fixed memory costs in stark contrast to
feedforward networks, whose memory requirements scale linearly with depth.
However, there is no free lunch -- backpropagation through implicit networks
often requires solving a costly Jacobian-based equation arising from the
implicit function theorem. We propose Jacobian-Free Backpropagation (JFB), a
fixed-memory approach that circumvents the need to solve Jacobian-based
equations. JFB makes implicit networks faster to train and significantly easier
to implement, without sacrificing test accuracy. Our experiments show implicit
networks trained with JFB are competitive with feedforward networks and prior
implicit networks given the same number of parameters.

本研究提出一种名为 Jacobian-Free Backpropagation 的算法，用于优化隐式网络的反向传播过程，以节省训练时间和实现难度，并在测试精度方面保持与前沿水平相当，同时仍能满足固定内存成本的需求。