Deep equilibrium (DEQ) models are widely recognized as a memory efficient
alternative to standard neural networks, achieving state-of-the-art performance
in language modeling and computer vision tasks. These models solve a fixed
point equation instead of explicitly computing the output, which sets them
apart from standard neural networks. However, existing DEQ models often lack
formal guarantees of the existence and uniqueness of the fixed point, and the
convergence of the numerical scheme used for computing the fixed point is not
formally established. As a result, DEQ models are potentially unstable in
practice. To address these drawbacks, we introduce a novel class of DEQ models
called positive concave deep equilibrium (pcDEQ) models. Our approach, which is
based on nonlinear Perron-Frobenius theory, enforces nonnegative weights and
activation functions that are concave on the positive orthant. By imposing
these constraints, we can easily ensure the existence and uniqueness of the
fixed point without relying on additional complex assumptions commonly found in
the DEQ literature, such as those based on monotone operator theory in convex
analysis. Furthermore, the fixed point can be computed with the standard fixed
point algorithm, and we provide theoretical guarantees of geometric
convergence, which, in particular, simplifies the training process. Experiments
demonstrate the competitiveness of our pcDEQ models against other implicit
models.

通过引入一种名为正凹深度平衡 (pcDEQ) 模型的新型 DEQ 模型类，通过基于非线性 Perron-Frobenius 理论的方法强制非负权重和激活函数，可以简化训练过程并且保证了固定点的存在和唯一性，进而解决了 DEQ 模型存在的固定点唯一性和收敛性问题，实验证明了 pcDEQ 模型在隐式模型中的竞争力。