This work presents BAdam, an optimizer that leverages the block coordinate optimization framework with Adam as the inner solver. BAdam offers a memory efficient approach to the full parameter finetuning of large language models and reduces running time of the backward process thanks to the chain rule property. Experimentally, we apply BAdam to instruction-tune the Llama 2-7B model on the Alpaca-GPT4 dataset using a single RTX3090-24GB GPU. The results indicate that BAdam exhibits superior convergence behavior in comparison to LoRA and LOMO. Furthermore, our downstream performance evaluation of the instruction-tuned models using the MT-bench shows that BAdam modestly surpasses LoRA and more substantially outperforms LOMO. Finally, we compare BAdam with Adam on a medium-sized task, i.e., finetuning RoBERTa-large on the SuperGLUE benchmark. The results demonstrate that BAdam is capable of narrowing the performance gap with Adam. Our code is available at https://github.com/Ledzy/BAdam.

该研究提出了BAdam，一种利用Adam作为内部求解器的块坐标优化框架的优化器。BAdam提供了一种内存高效的方法，用于对大型语言模型进行完全参数微调，通过链式规则属性减少了后向过程的运行时间。实验结果表明，BAdam在比较LoRA和LOMO时表现出优越的收敛行为。此外，我们通过对MT-bench的下游性能评估来评估经过指导调整的模型，结果表明BAdam略优于LoRA，并且在LOMO方面表现更好。最后，我们将BAdam与Adam在一个中等规模任务上进行了比较，即在SuperGLUE基准测试中对RoBERTa-large进行微调，结果表明BAdam能够缩小与Adam之间的性能差距。

BAdam: 大型语言模型的内存高效全参数训练方法