In this work, we introduce a novel paradigm called Simulated
Overparametrization (SOP). SOP merges the computational efficiency of compact
models with the advanced learning proficiencies of overparameterized models.
SOP proposes a unique approach to model training and inference, where a model
with a significantly larger number of parameters is trained in such a way that
a smaller, efficient subset of these parameters is used for the actual
computation during inference. Building upon this framework, we present a novel,
architecture agnostic algorithm called "majority kernels", which seamlessly
integrates with predominant architectures, including Transformer models.
Majority kernels enables the simulated training of overparameterized models,
resulting in performance gains across architectures and tasks. Furthermore, our
approach adds minimal overhead to the cost incurred (wall clock time) at
training time. The proposed approach shows strong performance on a wide variety
of datasets and models, even outperforming strong baselines such as
combinatorial optimization methods based on submodular optimization.

在这项研究中，我们介绍了一种名为模拟超参数化（SOP）的新范式。SOP 将紧凑模型的计算效率与超参数化模型的先进学习能力相结合，提出了一种独特的模型训练和推断方法。我们提出了一种与主流架构（包括 Transformer 模型）无缝集成的架构无关算法 “majority kernels”，该算法使得模拟超参数化模型的训练，在各种数据集和模型上表现出强大的性能，并在训练时间中增加了最小的开销。