Implicit models such as Deep Equilibrium Models (DEQs) have garnered
significant attention in the community for their ability to train infinite
layer models with elegant solution-finding procedures and constant memory
footprint. However, despite several attempts, these methods are heavily
constrained by model inefficiency and optimization instability. Furthermore,
fair benchmarking across relevant methods for vision tasks is missing. In this
work, we revisit the line of implicit models and trace them back to the
original weight-tied models. Surprisingly, we observe that weight-tied models
are more effective, stable, as well as efficient on vision tasks, compared to
the DEQ variants. Through the lens of these simple-yet-clean weight-tied
models, we further study the fundamental limits in the model capacity of such
models and propose the use of distinct sparse masks to improve the model
capacity. Finally, for practitioners, we offer design guidelines regarding the
depth, width, and sparsity selection for weight-tied models, and demonstrate
the generalizability of our insights to other learning paradigms.

本研究重访了隐式模型的路线并将其追溯到最初的权重绑定模型，并提出使用不同的稀疏掩码来改善模型容量，在权重绑定模型中提供有关深度、宽度和稀疏选择的设计指导，并展示了我们的观察结果对其他学习范例的适用性。

隐式模型再探：Weight-tied 模型在视觉任务中的稀疏性能权衡

Revisiting Implicit Models: Sparsity Trade-offs Capability in  Weight-tied Model for Vision Tasks

Continual Learning (CL) methods mainly focus on avoiding catastrophic
forgetting and learning representations that are transferable to new tasks.
Recently, Wortsman et al. (2020) proposed a CL method, SupSup, which uses a
randomly initialized, fixed base network (model) and finds a supermask for each
new task that selectively keeps or removes each weight to produce a subnetwork.
They prevent forgetting as the network weights are not being updated. Although
there is no forgetting, the performance of the supermask is sub-optimal because
fixed weights restrict its representational power. Furthermore, there is no
accumulation or transfer of knowledge inside the model when new tasks are
learned. Hence, we propose ExSSNeT (Exclusive Supermask SubNEtwork Training),
which performs exclusive and non-overlapping subnetwork weight training. This
avoids conflicting updates to the shared weights by subsequent tasks to improve
performance while still preventing forgetting. Furthermore, we propose a novel
KNN-based Knowledge Transfer (KKT) module that dynamically initializes a new
task's mask based on previous tasks for improving knowledge transfer. We
demonstrate that ExSSNeT outperforms SupSup and other strong previous methods
on both text classification and vision tasks while preventing forgetting.
Moreover, ExSSNeT is particularly advantageous for sparse masks that activate
2-10% of the model parameters, resulting in an average improvement of 8.3% over
SupSup. Additionally, ExSSNeT scales to a large number of tasks (100), and our
KKT module helps to learn new tasks faster while improving overall performance.
Our code is available at this https URL

通过 exclusive subnetwork training 和 KNN-based knowledge transfer 方法，我们提出了 ExSSNeT 模型，避免遗忘同时实现新任务的快速学习和知识转移，在文本分类和视觉任务中得到了比 SupSup 更好的结果。