Transferring the reasoning capability from stronger large language models
(LLMs) to smaller ones has been quite appealing, as smaller LLMs are more
flexible to deploy with less expense. Among the existing solutions, knowledge
distillation stands out due to its outstanding efficiency and generalization.
However, existing methods suffer from several drawbacks, including limited
knowledge diversity and the lack of rich contextual information. To solve the
problems and facilitate the learning of compact language models, we propose
TinyLLM, a novel knowledge distillation paradigm to learn a small student LLM
from multiple large teacher LLMs. In particular, we encourage the student LLM
to not only generate the correct answers but also understand the rationales
behind these answers. Given that different LLMs possess diverse reasoning
skills, we guide the student model to assimilate knowledge from various teacher
LLMs. We further introduce an in-context example generator and a
teacher-forcing Chain-of-Thought strategy to ensure that the rationales are
accurate and grounded in contextually appropriate scenarios. Extensive
experiments on six datasets across two reasoning tasks demonstrate the
superiority of our method. Results show that TinyLLM can outperform large
teacher LLMs significantly, despite having a considerably smaller model size.

通过知识蒸馏，从多个大型教师语言模型中学习小型学生语言模型 TinyLLM，以解决当前方法存在的知识多样性有限和缺乏丰富的上下文信息等问题，并通过引入上下文示例生成器和 teacher-forcing 链式推理策略来确保合理的推理基于适当情境，从而在两个推理任务的六个数据集上的广泛实验中展示了 TinyLLM 方法的优越性，结果表明尽管模型尺寸较小，但 TinyLLM 可显著优于大型教师语言模型。