We propose that small pretrained foundational generative language models with
millions of parameters can be utilized as a general learning framework for
sequence-based tasks. Our proposal overcomes the computational resource, skill
set, and timeline challenges associated with training neural networks and
language models from scratch. Further, our approach focuses on creating small
and highly specialized models that can accurately execute a challenging task of
which the base model is incapable of performing. We demonstrate that 125M,
350M, and 1.3B parameter pretrained foundational language models can be
instruction fine-tuned with 10,000-to-1,000,000 instruction examples to achieve
near state-of-the-art results on challenging cheminformatics tasks. We also
demonstrate the role of successive language model fine-tuning epochs on
improved outcomes, as well as the importance of both data formatting and
pretrained foundational language model selection for instruction fine-tuning
success.

我们提出，可以利用具有数百万参数的小型预训练基础生成性语言模型作为一种通用学习框架来处理基于序列的任务。我们的方法克服了从头开始训练神经网络和语言模型所面临的计算资源、技能和时间线挑战。此外，我们的方法专注于创建能够准确执行基础模型无法完成的挑战性任务的小型高度专业化模型。我们证明了使用 125M、350M 和 1.3B 参数的预训练基础语言模型可以通过 10,000 到 1,000,000 个指令示例进行指令微调，从而在具有挑战性的化学信息学任务上取得接近最先进的结果。我们还展示了连续语言模型微调周期对改善结果的作用，以及数据格式化和预训练基础语言模型选择对指令微调成功的重要性。