Implicit deep learning has recently gained popularity with applications
ranging from meta-learning to Deep Equilibrium Networks (DEQs). In its general
formulation, it relies on expressing some components of deep learning pipelines
implicitly, typically via a root equation called the inner problem. In
practice, the solution of the inner problem is approximated during training
with an iterative procedure, usually with a fixed number of inner iterations.
During inference, the inner problem needs to be solved with new data. A popular
belief is that increasing the number of inner iterations compared to the one
used during training yields better performance. In this paper, we question such
an assumption and provide a detailed theoretical analysis in a simple setting.
We demonstrate that overparametrization plays a key role: increasing the number
of iterations at test time cannot improve performance for overparametrized
networks. We validate our theory on an array of implicit deep-learning
problems. DEQs, which are typically overparametrized, do not benefit from
increasing the number of iterations at inference while meta-learning, which is
typically not overparametrized, benefits from it.

本文提出了问题，并在简单的情境下给出了详细的理论分析，证明了对于超参数化网络，与训练时相比增加测试时的内部迭代次数不能改善性能。

隐式深度学习中像训练测试

Test like you Train in Implicit Deep Learning

Deep equilibrium networks (DEQs) are a new class of models that eschews
traditional depth in favor of finding the fixed point of a single nonlinear
layer. These models have been shown to achieve performance competitive with the
state-of-the-art deep networks while using significantly less memory. Yet they
are also slower, brittle to architectural choices, and introduce potential
instability to the model. In this paper, we propose a regularization scheme for
DEQ models that explicitly regularizes the Jacobian of the fixed-point update
equations to stabilize the learning of equilibrium models. We show that this
regularization adds only minimal computational cost, significantly stabilizes
the fixed-point convergence in both forward and backward passes, and scales
well to high-dimensional, realistic domains (e.g., WikiText-103 language
modeling and ImageNet classification). Using this method, we demonstrate, for
the first time, an implicit-depth model that runs with approximately the same
speed and level of performance as popular conventional deep networks such as
ResNet-101, while still maintaining the constant memory footprint and
architectural simplicity of DEQs. Code is available at
this https URL .

本文提出一种正则化方案来加强深度均衡网络（DEQ）模型的学习稳定性，该正则化方案显著提高了 DEQ 模型的收敛速度和性能，使得 DEQ 模型与传统深度网络在速度和性能上保持相当，并且具有恒定的内存占用和简单的架构。

通过雅各比正则化稳定平衡模型

Stabilizing Equilibrium Models by Jacobian Regularization

Implicit-depth models such as Deep Equilibrium Networks have recently been
shown to match or exceed the performance of traditional deep networks while
being much more memory efficient. However, these models suffer from unstable
convergence to a solution and lack guarantees that a solution exists. On the
other hand, Neural ODEs, another class of implicit-depth models, do guarantee
existence of a unique solution but perform poorly compared with traditional
networks. In this paper, we develop a new class of implicit-depth model based
on the theory of monotone operators, the Monotone Operator Equilibrium Network
(monDEQ). We show the close connection between finding the equilibrium point of
an implicit network and solving a form of monotone operator splitting problem,
which admits efficient solvers with guaranteed, stable convergence. We then
develop a parameterization of the network which ensures that all operators
remain monotone, which guarantees the existence of a unique equilibrium point.
Finally, we show how to instantiate several versions of these models, and
implement the resulting iterative solvers, for structured linear operators such
as multi-scale convolutions. The resulting models vastly outperform the Neural
ODE-based models while also being more computationally efficient. Code is
available at this http URL

本文开发了一种基于单调算子理论的新类别的内部层数模型，即单调算子平衡网络（monDEQ），通过带有保证和稳定收敛的高效求解器来解决难以稳定收敛和缺乏唯一解保证的传统的内部层数问题，该模型的参数化能确保所有算子保持单调性，从而保证存在独特的平衡点，并在多尺度卷积等结构化线性算子上实现了多个版本的这些模型，性能均明显优于基于神经 ODE 的模型，且更加高效。