Recent work on explainable NLP has shown that few-shot prompting can enable
large pretrained language models (LLMs) to generate grammatical and factual
natural language explanations for data labels. In this work, we study the
connection between explainability and sample hardness by investigating the
following research question - "Are LLMs and humans equally good at explaining
data labels for both easy and hard samples?" We answer this question by first
collecting human-written explanations in the form of generalizable commonsense
rules on the task of Winograd Schema Challenge (Winogrande dataset). We compare
these explanations with those generated by GPT-3 while varying the hardness of
the test samples as well as the in-context samples. We observe that (1) GPT-3
explanations are as grammatical as human explanations regardless of the
hardness of the test samples, (2) for easy examples, GPT-3 generates highly
supportive explanations but human explanations are more generalizable, and (3)
for hard examples, human explanations are significantly better than GPT-3
explanations both in terms of label-supportiveness and generalizability
judgements. We also find that hardness of the in-context examples impacts the
quality of GPT-3 explanations. Finally, we show that the supportiveness and
generalizability aspects of human explanations are also impacted by sample
hardness, although by a much smaller margin than models. Supporting code and
data are available at this https URL

通过人类编写的常识规则与 GPT-3 生成的描述进行比较，本研究研究了 LLM、人类在简单与困难例子的数据标注解释方面的能力，研究结果表明对于简单示例，GPT-3 生成的解释具有高度的支持性，而人类解释更具有通用性，对于困难示例，人类解释在标签支持和通用判断方面都比 GPT-3 解释更好。

难例也更难解释吗？人类和模型生成的解释研究

Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations

Ensemble knowledge distillation can extract knowledge from multiple teacher
models and encode it into a single student model. Many existing methods learn
and distill the student model on labeled data only. However, the teacher models
are usually learned on the same labeled data, and their predictions have high
correlations with groudtruth labels. Thus, they cannot provide sufficient
knowledge complementary to task labels for student teaching. Distilling on
unseen unlabeled data has the potential to enhance the knowledge transfer from
the teachers to the student. In this paper, we propose a unified and effective
ensemble knowledge distillation method that distills a single student model
from an ensemble of teacher models on both labeled and unlabeled data. Since
different teachers may have diverse prediction correctness on the same sample,
on labeled data we weight the predictions of different teachers according to
their correctness. In addition, we weight the distillation loss based on the
overall prediction correctness of the teacher ensemble to distill high-quality
knowledge. On unlabeled data, there is no groundtruth to evaluate prediction
correctness. Fortunately, the disagreement among teachers is an indication of
sample hardness, and thereby we weight the distillation loss based on teachers'
disagreement to emphasize knowledge distillation on important samples.
Extensive experiments on four datasets show the effectiveness of our proposed
ensemble distillation method.

本文提出了一种有效的集成知识蒸馏方法，该方法能够从多个教师模型学习未标记数据的知识，并据此训练单个学生模型。研究表明，通过考虑不同教师之间的预测差异以及样本难度，可以进一步提高蒸馏的效果。