In this paper, we employ the mathematical framework on Transformers developed by \citet{sander2022sinkformers,geshkovski2023emergence,geshkovski2023mathematical} to explore how variations in attention parameters and initial token values impact the structural dynamics of token clusters. Our analysis demonstrates that while the clusters within a modified attention matrix dynamics can exhibit significant divergence from the original over extended periods, they maintain close similarities over shorter intervals, depending on the parameter differences. This work contributes to the fine-tuning field through practical applications to the LoRA algorithm \cite{hu2021lora,peft}, enhancing our understanding of the behavior of LoRA-enhanced Transformer models.

本文利用由Sander等人（2022）和Geshkovski等人（2023）开发的Transformer数学框架，研究了注意力参数和初始令牌值的变化对令牌群集的结构动态的影响。我们的分析表明，虽然修改后的注意力矩阵动力学中的群集可以在较长时间内与原始群集显著分歧，但在较短时间内，它们在参数差异上保持较高的相似性。这项工作通过在LoRA算法上的实际应用（Hu等人，2021；PEFT）对微调领域做出了贡献，增进了我们对LoRA增强Transformer模型行为的理解。

LoRA对Transformers中聚类群的影响