Parameter generation has emerged as a novel paradigm for neural network development, offering an alternative to traditional neural network training by synthesizing high-quality model weights directly. In the context of Low-Rank Adaptation (LoRA) for evolving ($\textit{i.e.}$, constantly updated) large language models (LLMs), this approach promises efficient adaptation without costly retraining. However, existing methods face critical limitations in simultaneously achieving scalability and controllability. In this paper, we introduce $\texttt{ORAL}$, a novel $\textbf{conditional recurrent diffusion}$ framework that addresses these challenges. $\texttt{ORAL}$ incorporates a novel conditioning mechanism that integrates model architecture and textual task specifications, enabling the generation of task-specific LoRA parameters that can seamlessly transfer across evolving foundation models. Our approach successfully scales to billions-of-parameter LLMs and maintains controllability. Through extensive experiments across seven language tasks, four vision tasks, and three multimodal tasks using five pre-trained LLMs, we demonstrate that $\texttt{ORAL}$ generates high-quality LoRA parameters that achieve comparable or superior performance to vanilla trained counterparts.

本研究解决了大语言模型（LLM）中低秩适应（LoRA）方法在可扩展性和可控性方面的关键局限。我们提出的条件递归扩散框架ORAL，通过结合模型架构和任务规格，能够生成适应特定任务的LoRA参数，实现高效的模型适应。实验结果表明，ORAL在多个语言、视觉和多模态任务中生成的LoRA参数可与传统训练的结果相媲美或优于其表现。

ORAL：通过条件递归扩散提升您的大规模LoRA模型