Transferring the reasoning capability from stronger large language models
(LLMs) to smaller ones has been quite appealing, as smaller LLMs are more
flexible to deploy with less expense. Among the existing solutions, knowledge
distillation stands out due to its outstanding efficiency and generalization.
However, existing methods suffer from several drawbacks, including limited
knowledge diversity and the lack of rich contextual information. To solve the
problems and facilitate the learning of compact language models, we propose
TinyLLM, a novel knowledge distillation paradigm to learn a small student LLM
from multiple large teacher LLMs. In particular, we encourage the student LLM
to not only generate the correct answers but also understand the rationales
behind these answers. Given that different LLMs possess diverse reasoning
skills, we guide the student model to assimilate knowledge from various teacher
LLMs. We further introduce an in-context example generator and a
teacher-forcing Chain-of-Thought strategy to ensure that the rationales are
accurate and grounded in contextually appropriate scenarios. Extensive
experiments on six datasets across two reasoning tasks demonstrate the
superiority of our method. Results show that TinyLLM can outperform large
teacher LLMs significantly, despite having a considerably smaller model size.

通过知识蒸馏，从多个大型教师语言模型中学习小型学生语言模型 TinyLLM，以解决当前方法存在的知识多样性有限和缺乏丰富的上下文信息等问题，并通过引入上下文示例生成器和 teacher-forcing 链式推理策略来确保合理的推理基于适当情境，从而在两个推理任务的六个数据集上的广泛实验中展示了 TinyLLM 方法的优越性，结果表明尽管模型尺寸较小，但 TinyLLM 可显著优于大型教师语言模型。

TinyLLM: 从多个大型语言模型中学习一个小型学生

TinyLLM: Learning a Small Student from Multiple Large Language Models

Conversational question-answering (CQA) systems aim to create interactive
search systems that effectively retrieve information by interacting with users.
To replicate human-to-human conversations, existing work uses human annotators
to play the roles of the questioner (student) and the answerer (teacher).
Despite its effectiveness, challenges exist as human annotation is
time-consuming, inconsistent, and not scalable. To address this issue and
investigate the applicability of large language models (LLMs) in CQA
simulation, we propose a simulation framework that employs zero-shot learner
LLMs for simulating teacher-student interactions. Our framework involves two
LLMs interacting on a specific topic, with the first LLM acting as a student,
generating questions to explore a given search topic. The second LLM plays the
role of a teacher by answering questions and is equipped with additional
information, including a text on the given topic. We implement both the student
and teacher by zero-shot prompting the GPT-4 model. To assess the effectiveness
of LLMs in simulating CQA interactions and understand the disparities between
LLM- and human-generated conversations, we evaluate the simulated data from
various perspectives. We begin by evaluating the teacher's performance through
both automatic and human assessment. Next, we evaluate the performance of the
student, analyzing and comparing the disparities between questions generated by
the LLM and those generated by humans. Furthermore, we conduct extensive
analyses to thoroughly examine the LLM performance by benchmarking
state-of-the-art reading comprehension models on both datasets. Our results
reveal that the teacher LLM generates lengthier answers that tend to be more
accurate and complete. The student LLM generates more diverse questions,
covering more aspects of a given topic.

对话问答系统使用大规模语言模型进行模拟和交互，提出了一种基于零 - shot 学习的模拟框架，通过教师 - 学生互动模式实现指定主题下的问题和回答生成。通过评估教师和学生模型的性能并比较人类生成对话与模型生成对话的差异，揭示了使用语言模型进行 CQA 交互模拟的有效性和表现。