There is increasing interest in distilling task-specific knowledge from large
language models (LLM) to smaller student models. Nonetheless, LLM distillation
presents a dual challenge: 1) there is a high cost associated with querying the
teacher LLM, such as GPT-4, for gathering an ample number of demonstrations; 2)
the teacher LLM might provide imperfect outputs with a negative impact on the
student's learning process. To enhance sample efficiency within
resource-constrained, imperfect teacher scenarios, we propose a three-component
framework leveraging three signal types. The first signal is the student's
self-consistency (consistency of student multiple outputs), which is a proxy of
the student's confidence. Specifically, we introduce a ``teaching assistant''
(TA) model to assess the uncertainty of both the student's and the teacher's
outputs via confidence scoring, which serves as another two signals for student
training. Furthermore, we propose a two-stage training schema to first warm up
the student with a small proportion of data to better utilize student's signal.
Experiments have shown the superiority of our proposed framework for four
complex reasoning tasks. On average, our proposed two-stage framework brings a
relative improvement of up to 20.79% compared to fine-tuning without any
signals across datasets.

通过提出一个三组件框架，利用自洽性、鉴别性和教师的不确定性作为学生培训的三种信号，我们在资源受限、教师性能不完美的场景中提高了样本效率。实验证明，相较于没有任何信号进行微调的情况，我们提出的两阶段框架平均带来了 20.79% 左右的相对提升，适用于四个复杂推理任务。