Fine-tuning large language models (LLMs) with classic first-order optimizers
entails prohibitive GPU memory due to the backpropagation process. Recent works
have turned to zeroth-order optimizers for fine-tuning, which save substantial
memory by using two forward passes. However, these optimizers are plagued by
the heterogeneity of parameter curvatures across different dimensions. In this
work, we propose HiZOO, a diagonal Hessian informed zeroth-order optimizer
which is the first work to leverage the diagonal Hessian to enhance
zeroth-order optimizer for fine-tuning LLMs. What's more, HiZOO avoids the
expensive memory cost and only increases one forward pass per step. Extensive
experiments on various models (350M~66B parameters) indicate that HiZOO
improves model convergence, significantly reducing training steps and
effectively enhancing model accuracy. Moreover, we visualize the optimization
trajectories of HiZOO on test functions, illustrating its effectiveness in
handling heterogeneous curvatures. Lastly, we provide theoretical proofs of
convergence for HiZOO. Code is publicly available at
this https URL

使用对角度黑塞矩阵增强零阶优化算法以提高大型语言模型的微调效果，显著减少训练步骤并有效增强模型准确性。

LLM 模型的无痛二阶微调：基于黑塞矩阵的零阶优化器

Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed  Zeroth-Order Optimizer

Fine-tuning language models (LMs) has yielded success on diverse downstream
tasks, but as LMs grow in size, backpropagation requires a prohibitively large
amount of memory. Zeroth-order (ZO) methods can in principle estimate gradients
using only two forward passes but are theorized to be catastrophically slow for
optimizing large models. In this work, we propose a memory-efficient
zerothorder optimizer (MeZO), adapting the classical ZO-SGD method to operate
in-place, thereby fine-tuning LMs with the same memory footprint as inference.
For example, with a single A100 80GB GPU, MeZO can train a 30-billion parameter
model, whereas fine-tuning with backpropagation can train only a 2.7B LM with
the same budget. We conduct comprehensive experiments across model types
(masked and autoregressive LMs), model scales (up to 66B), and downstream tasks
(classification, multiple-choice, and generation). Our results demonstrate that
(1) MeZO significantly outperforms in-context learning and linear probing; (2)
MeZO achieves comparable performance to fine-tuning with backpropagation across
multiple tasks, with up to 12x memory reduction; (3) MeZO is compatible with
both full-parameter and parameter-efficient tuning techniques such as LoRA and
prefix tuning; (4) MeZO can effectively optimize non-differentiable objectives
(e.g., maximizing accuracy or F1). We support our empirical findings with
theoretical insights, highlighting how adequate pre-training and task prompts
enable MeZO to fine-tune huge models, despite classical ZO analyses suggesting
otherwise.

本篇论文提出了一种名为 MeZO 的零阶优化方法，以替代反向传播法来进行大规模语言模型的微调，从而实现与推理阶段相同的内存占用，该方法在理论上被认为优化大型模型的速度极慢，但本研究的实验结果表明，MeZO 能够使用少至单个 A100 80GB GPU 就能训练出具有几十亿甚至数百亿参数规模的语言模型，并在多个下游任务上与反向传播微调的性能相当。