This work presents BAdam, an optimizer that leverages the block coordinate
optimization framework with Adam as the inner solver. BAdam offers a memory
efficient approach to the full parameter finetuning of large language models
and reduces running time of the backward process thanks to the chain rule
property. Experimentally, we apply BAdam to instruction-tune the Llama 2-7B
model on the Alpaca-GPT4 dataset using a single RTX3090-24GB GPU. The results
indicate that BAdam exhibits superior convergence behavior in comparison to
LoRA and LOMO. Furthermore, our downstream performance evaluation of the
instruction-tuned models using the MT-bench shows that BAdam modestly surpasses
LoRA and more substantially outperforms LOMO. Finally, we compare BAdam with
Adam on a medium-sized task, i.e., finetuning RoBERTa-large on the SuperGLUE
benchmark. The results demonstrate that BAdam is capable of narrowing the
performance gap with Adam. Our code is available at
this https URL

该研究提出了 BAdam，一种利用 Adam 作为内部求解器的块坐标优化框架的优化器。BAdam 提供了一种内存高效的方法，用于对大型语言模型进行完全参数微调，通过链式规则属性减少了后向过程的运行时间。实验结果表明，BAdam 在比较 LoRA 和 LOMO 时表现出优越的收敛行为。此外，我们通过对 MT-bench 的下游性能评估来评估经过指导调整的模型，结果表明 BAdam 略优于 LoRA，并且在 LOMO 方面表现更好。最后，我们将 BAdam 与 Adam 在一个中等规模任务上进行了比较，即在 SuperGLUE 基准测试中对 RoBERTa-large 进行微调，结果表明 BAdam 能够缩小与 Adam 之间的性能差距。