Bilevel optimization has shown its utility across various machine learning settings, yet most algorithms in practice require second-order information, making it challenging to scale them up. Only recently, a paradigm of first-order algorithms emerged, capable of effectively addressing bilevel optimization problems. Nevertheless, the practical efficiency of this paradigm remains unverified, particularly in the context of large language models (LLMs). This paper introduces the first scalable instantiation of this paradigm called ScaleBiO, focusing on bilevel optimization for large-scale LLM data reweighting. By combining with a recently proposed memory-efficient training technique called LISA, our novel algorithm allows the paradigm to scale to 34-billion-parameter LLMs on eight A40 GPUs, marking the first successful application of bilevel optimization under practical scenarios for large-sized LLMs. Empirically, extensive experiments on data reweighting verify the effectiveness of ScaleBiO for different-scaled models, including GPT-2, LLaMA-3-8B, GPT-NeoX-20B, and Yi-34B, where bilevel optimization succeeds in filtering irrelevant data samples and selecting informative samples. Theoretically, ScaleBiO ensures the optimality of the learned data weights, along with a convergence guarantee matching the conventional first-order bilevel optimization paradigm on smooth and strongly convex objectives.

本研究提出了第一种可扩展实例，称为ScaleBiO，通过与一种称为LISA的新型算法相结合，使软件在八个A40 GPU上扩展到34亿参数的大型语言模型（LLM），标志着在实际应用场景中首次成功应用双层优化于大型LLM。从实证上来说，对于不同规模的模型，包括GPT-2，LLaMA-3-8B，GPT-NeoX-20B和Yi-34B，通过数据重加权广泛实验验证了ScaleBiO的有效性，其中双层优化成功地过滤了无关的数据样本，并选择了信息量较大的样本。从理论上来说，ScaleBiO确保了学习到的数据权重的最优性，并具有匹配传统的一阶平滑凸优化双层优化范式的收敛保证。

ScaleBiO: 面向LLM数据重新加权的可扩展的双层优化