Large models represent a groundbreaking advancement in multiple application
fields, enabling remarkable achievements across various tasks. However, their
unprecedented scale comes with significant computational costs. These models,
often consisting of billions of parameters, require vast amounts of
computational resources for execution. Especially, the expansive scale and
computational demands pose considerable challenges when customizing them for
particular downstream tasks, particularly over the hardware platforms
constrained by computational capabilities. Parameter Efficient Fine-Tuning
(PEFT) provides a practical solution by efficiently adapt the large models over
the various downstream tasks. In particular, PEFT refers to the process of
adjusting the parameters of a pre-trained large models to adapt it to a
specific task while minimizing the number of additional parameters introduced
or computational resources required. This approach is particularly important
when dealing with large language models with high parameter counts, as
fine-tuning these models from scratch can be computationally expensive and
resource-intensive, posing considerable challenges in the supporting system
platform design. In this survey, we present comprehensive studies of various
PEFT algorithms, examining their performance and computational overhead.
Moreover, we provide an overview of applications developed using different PEFT
algorithms and discuss common techniques employed to mitigate computation costs
for PEFT. In addition to the algorithmic perspective, we overview various
real-world system designs to investigate the implementation costs associated
with different PEFT algorithms. This survey serves as an indispensable resource
for researchers aiming to understand both the PEFT algorithm and its system
implementation, offering detailed insights into recent advancements and
practical applications.

通过介绍 Parameter Efficient Fine-Tuning 算法以及系统实现相关内容，该论文对大模型在计算成本方面的问题进行了综述，提供了对性能和系统实施的深入洞察，为研究人员了解最新发展和实际应用提供了不可或缺的资源。

大型模型的参数高效微调：综合调查

Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

Extensive studies have shown that deep learning models are vulnerable to
adversarial and natural noises, yet little is known about model robustness on
noises caused by different system implementations. In this paper, we for the
first time introduce SysNoise, a frequently occurred but often overlooked noise
in the deep learning training-deployment cycle. In particular, SysNoise happens
when the source training system switches to a disparate target system in
deployments, where various tiny system mismatch adds up to a non-negligible
difference. We first identify and classify SysNoise into three categories based
on the inference stage; we then build a holistic benchmark to quantitatively
measure the impact of SysNoise on 20+ models, comprehending image
classification, object detection, instance segmentation and natural language
processing tasks. Our extensive experiments revealed that SysNoise could bring
certain impacts on model robustness across different tasks and common
mitigations like data augmentation and adversarial training show limited
effects on it. Together, our findings open a new research topic and we hope
this work will raise research attention to deep learning deployment systems
accounting for model performance. We have open-sourced the benchmark and
framework at this https URL

本文对深度学习模型在不同系统实现中的鲁棒性问题进行了研究，并首次引入了系统噪声（SysNoise）这一概念，该噪声在深度学习训练和部署过程中的影响不容忽视，作者进一步对其进行分类并构建了综合基准来量化其对模型性能的影响，发现 SysNoise 会对模型的鲁棒性产生一定影响，但常见的数据增强和对抗性训练等方法对其起到的缓解作用有限，从而提出了对深度学习部署系统的性能考虑是一个新的研究领域。