While large language models (LLMs) often adopt finetuning to unlock their
capabilities for downstream applications, our understanding on the inductive
biases (especially the scaling properties) of different finetuning methods is
still limited. To fill this gap, we conduct systematic experiments studying
whether and how different scaling factors, including LLM model size,
pretraining data size, new finetuning parameter size and finetuning data size,
affect the finetuning performance. We consider two types of finetuning --
full-model tuning (FMT) and parameter efficient tuning (PET, including prompt
tuning and LoRA), and explore their scaling behaviors in the data-limited
regime where the LLM model size substantially outweighs the finetuning data
size. Based on two sets of pretrained bilingual LLMs from 1B to 16B and
experiments on bilingual machine translation and multilingual summarization
benchmarks, we find that 1) LLM finetuning follows a powerbased multiplicative
joint scaling law between finetuning data size and each other scaling factor;
2) LLM finetuning benefits more from LLM model scaling than pretraining data
scaling, and PET parameter scaling is generally ineffective; and 3) the optimal
finetuning method is highly task- and finetuning data-dependent. We hope our
findings could shed light on understanding, selecting and developing LLM
finetuning methods.

大语言模型的微调方法的缩放因子对模型性能的影响的系统实验结果表明，LLM finetuning 遵循微调数据量与其他缩放因子之间的幂函数乘法联合缩放规律，LLM 模型尺寸的扩大比预训练数据尺寸的扩大对 finetuning 更有益处，而仅考虑参数缩放的效果通常并不明显，同时最优的 finetuning 方法与任务和微调数据相关联，这些结果对于选择和开发 LLM finetuning 方法具有指导意义。

当扩展与 LLM 微调相遇：数据、模型和微调方法的影响

When Scaling Meets LLM Finetuning: The Effect of Data, Model and  Finetuning Method

With the rapid scaling of large language models (LLMs), serving numerous
LoRAs concurrently has become increasingly impractical, leading to unaffordable
costs and necessitating more parameter-efficient finetuning methods. In this
work, we introduce Partially Rotation-enhanced Low-Rank Adaptation (PRoLoRA),
an intra-layer sharing mechanism comprising four essential components:
broadcast reduction, rotation enhancement, partially-sharing refinement, and
rectified initialization strategy. As a superset of LoRA, PRoLoRA pertains its
advantages, and effectively circumvent the drawbacks of peer parameter-sharing
methods with superior model capacity, practical feasibility, and broad
applicability. Empirical experiments demonstrate the remarkably higher
parameter efficiency of PRoLoRA in both specific parameter budget and
performance target scenarios, and its scalability to larger LLMs. Notably, with
one time less trainable parameters, PRoLoRA still outperforms LoRA on multiple
instruction tuning datasets. Subsequently, an ablation study is conducted to
validate the necessity of individual components and highlight the superiority
of PRoLoRA over three potential variants. Hopefully, the conspicuously higher
parameter efficiency can establish PRoLoRA as a resource-friendly alternative
to LoRA.

通过引入部分旋转增强的低秩适应机制 (PR0LoRA)，本文在大型语言模型 (Large Language Models) 领域提出了一种参数效率更高、适用范围更广的新方法，实验证明 PRoLoRA 在特定参数预算和性能目标场景下具有明显更高的参数效率，并且扩展性更强。希望 PRoLoRA 能作为一种资源友好的替代方法来使用，优于 LoRA 方法。

PRoLoRA： Partial Rotation 强化更高效的参数化 LoRA

PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA

Automation is one of the cornerstones of contemporary material discovery.
Bayesian optimization (BO) is an essential part of such workflows, enabling
scientists to leverage prior domain knowledge into efficient exploration of a
large molecular space. While such prior knowledge can take many forms, there
has been significant fanfare around the ancillary scientific knowledge
encapsulated in large language models (LLMs). However, existing work thus far
has only explored LLMs for heuristic materials searches. Indeed, recent work
obtains the uncertainty estimate -- an integral part of BO -- from
point-estimated, non-Bayesian LLMs. In this work, we study the question of
whether LLMs are actually useful to accelerate principled Bayesian optimization
in the molecular space. We take a sober, dispassionate stance in answering this
question. This is done by carefully (i) viewing LLMs as fixed feature
extractors for standard but principled BO surrogate models and by (ii)
leveraging parameter-efficient finetuning methods and Bayesian neural networks
to obtain the posterior of the LLM surrogate. Our extensive experiments with
real-world chemistry problems show that LLMs can be useful for BO over
molecules, but only if they have been pretrained or finetuned with
domain-specific data.

大型语言模型在分子空间中进行 Bayesian 优化的实验表明，只有通过预训练或领域特定数据的微调，才能使 LLMs 对于分子的 Bayesian 优化具有实用性。