This work studies the neural tangent kernel (NTK) of the deep equilibrium
(DEQ) model, a practical ``infinite-depth'' architecture which directly
computes the infinite-depth limit of a weight-tied network via root-finding.
Even though the NTK of a fully-connected neural network can be stochastic if
its width and depth both tend to infinity simultaneously, we show that
contrarily a DEQ model still enjoys a deterministic NTK despite its width and
depth going to infinity at the same time under mild conditions. Moreover, this
deterministic NTK can be found efficiently via root-finding.

这项研究探讨了深度平衡模型的神经切向核（NTK），一种实用的 “无限深度” 架构，通过根求解直接计算出与之相关联的网络的无限深度极限。尽管完全连接的神经网络的 NTK 在宽度和深度同时趋于无穷大时可能是随机的，但我们证明在一些温和条件下，DEQ 模型仍然具有确定性 NTK。此外，可以通过根求解高效地找到这种确定性 NTK。

关于平衡模型的神经切向核

On the Neural Tangent Kernel of Equilibrium Models

We present a new approach to modeling sequential data: the deep equilibrium
model (DEQ). Motivated by an observation that the hidden layers of many
existing deep sequence models converge towards some fixed point, we propose the
DEQ approach that directly finds these equilibrium points via root-finding.
Such a method is equivalent to running an infinite depth (weight-tied)
feedforward network, but has the notable advantage that we can analytically
backpropagate through the equilibrium point using implicit differentiation.
Using this approach, training and prediction in these networks require only
constant memory, regardless of the effective "depth" of the network. We
demonstrate how DEQs can be applied to two state-of-the-art deep sequence
models: self-attention transformers and trellis networks. On large-scale
language modeling tasks, such as the WikiText-103 benchmark, we show that DEQs
1) often improve performance over these state-of-the-art models (for similar
parameter counts); 2) have similar computational requirements to existing
models; and 3) vastly reduce memory consumption (often the bottleneck for
training large sequence models), demonstrating an up-to 88% memory reduction in
our experiments. The code is available at this https URL .

本文介绍了一种新的应用于序列数据建模的方法 —— 深度平衡模型，并比较其在大规模语言模型任务上的性能，该方法可通过求解根来直接获取固定点，训练和预测所需的内存只需常数级别，大大减少了存储消耗。

深度平衡模型

Deep Equilibrium Models

Consider a process, stochastic or deterministic, obtained by using a
numerical integration scheme, or from Monte-Carlo methods involving an
approximation to an integral, or a Newton-Raphson iteration to approximate the
root of an equation. We will assume that we can sample from the distribution of
the process from time 0 to finite time n. We propose a scheme for unbiased
estimation of the limiting value of the process, together with estimates of
standard error and apply this to examples including numerical integrals,
root-finding and option pricing in a Heston Stochastic Volatility model. This
results in unbiased estimators in place of biased ones i nmany potential
applications.

提出了一个无偏的方案，用于估计过程的极限值，并将其应用于包括数值积分，根搜索和 Heston 随机波动性模型的期权定价的示例中，从而在许多潜在的应用中将有偏估计替换为无偏估计。