Applying a pre-trained large model to downstream tasks is prohibitive under resource-constrained conditions. Recent dominant approaches for addressing efficiency issues involve adding a few learnable parameters to the fixed backbone model. This strategy, however, leads to more challenges in loading large models for downstream fine-tuning with limited resources. In this paper, we propose a novel method for increasing the parameter efficiency of pre-trained models by introducing an intermediate pre-training stage. To this end, we first employ low-rank approximation to compress the original large model and then devise a feature distillation module and a weight perturbation regularization module. These modules are specifically designed to enhance the low-rank model. Concretely, we update only the low-rank model while freezing the backbone parameters during pre-training. This allows for direct and efficient utilization of the low-rank model for downstream tasks. The proposed method achieves both efficiencies in terms of required parameters and computation time while maintaining comparable results with minimal modifications to the base architecture. Specifically, when applied to three vision-only and one vision-language Transformer models, our approach often demonstrates a $\sim$0.6 point decrease in performance while reducing the original parameter size by 1/3 to 2/3.

通过引入中间的预训练阶段，我们提出了一种新方法来增加预训练模型的参数效率，该方法通过低秩逼近来压缩原始的大模型，并设计了特征蒸馏模块和权重扰动正则化模块，以增强低秩模型。这种方法在保持基本架构的最小修改的同时，在所需参数和计算时间方面同时实现了效率，通常将原始参数大小减少1/3至2/3，并降低约0.6个性能点。

PELA:学习具有低秩近似的参数高效模型