BriefGPT.xyz
Sep, 2023
在训练的哪个阶段,代码数据对LLMs的推理有帮助?
At Which Training Stage Does Cocde Data Help LLMs Reasoning?
HTML
PDF
Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang...
TL;DR
使用代码数据在预训练和指令调整阶段可以显著增强大型语言模型的推理能力,同时动态混合代码和文本数据有助于逐步学习推理能力。
Abstract
large language models
(LLMs) have exhibited remarkable reasoning capabilities and become the foundation of language technologies. Inspired by the great success of
code data
in training LLMs, we naturally wonder a
→