In this paper, we present an efficient method for storing fine-tuned models
by leveraging the low-rank properties of weight residuals. Our key observation
is that weight residuals in large overparameterized models exhibit even
stronger low-rank characteristics. Based on this insight, we propose Efficient
Residual Encoding (ERE), a novel approach that achieves efficient storage of
fine-tuned model weights by approximating the low-rank weight residuals.
Furthermore, we analyze the robustness of weight residuals and push the limit
of storage efficiency by utilizing additional quantization and layer-wise rank
allocation. Our experimental results demonstrate that our method significantly
reduces memory footprint while preserving performance in various tasks and
modalities. We release our code.

通过权重残差的低秩特性，本文提出了一种高效的存储 fine-tuned 模型的方法，称为 Efficient Residual Encoding（ERE），通过低秩权重残差的逼近来实现 fine-tuned 模型权重的高效存储，并通过使用额外的量化和分层秩分配来分析权重残差的健壮性并推进存储效率上限。实验结果表明，ERE 方法在各种任务和模态下显着减少内存占用并保持性能，代码已发布。