It is believed that in knowledge distillation (KD), the role of the teacher
is to provide an estimate for the unknown Bayes conditional probability
distribution (BCPD) to be used in the student training process. Conventionally,
this estimate is obtained by training the teacher using maximum log-likelihood
(MLL) method. To improve this estimate for KD, in this paper we introduce the
concept of conditional mutual information (CMI) into the estimation of BCPD and
propose a novel estimator called the maximum CMI (MCMI) method. Specifically,
in MCMI estimation, both the log-likelihood and CMI of the teacher are
simultaneously maximized when the teacher is trained. Through Eigen-CAM, it is
further shown that maximizing the teacher's CMI value allows the teacher to
capture more contextual information in an image cluster. Via conducting a
thorough set of experiments, we show that by employing a teacher trained via
MCMI estimation rather than one trained via MLL estimation in various
state-of-the-art KD frameworks, the student's classification accuracy
consistently increases, with the gain of up to 3.32\%. This suggests that the
teacher's BCPD estimate provided by MCMI method is more accurate than that
provided by MLL method. In addition, we show that such improvements in the
student's accuracy are more drastic in zero-shot and few-shot settings.
Notably, the student's accuracy increases with the gain of up to 5.72\% when
5\% of the training samples are available to the student (few-shot), and
increases from 0\% to as high as 84\% for an omitted class (zero-shot). The
code is available at https://github.com/iclr2024mcmi/ICLRMCMI.

通过引入条件互信息的概念以提高知识蒸馏中教师对未知贝叶斯条件概率分布的估计，本文提出了一种称为最大条件互信息方法的新估计器。通过对教师进行最大条件互信息估计，能够捕捉到图像聚类中更多的上下文信息，从而在各种先进的知识蒸馏框架中使用通过最大条件互信息估计而不是最大对数似然估计的教师，在学生的分类准确率上获得持续提高，增益高达 3.32％。此外，研究还表明，这种提高在零样本和少样本情况下更为显著，当学生只有 5％的训练样本可用（少样本）时，准确率提高了 5.72％，在省略类别的情况下（零样本），准确率从 0％增加到高达 84％。

基于条件互信息的贝叶斯条件分布估计用于知识蒸馏

Bayes Conditional Distribution Estimation for Knowledge Distillation  Based on Conditional Mutual Information

Coaching, which involves classroom observation and expert feedback, is a
widespread and fundamental part of teacher training. However, the majority of
teachers do not have access to consistent, high quality coaching due to limited
resources and access to expertise. We explore whether generative AI could
become a cost-effective complement to expert feedback by serving as an
automated teacher coach. In doing so, we propose three teacher coaching tasks
for generative AI: (A) scoring transcript segments based on classroom
observation instruments, (B) identifying highlights and missed opportunities
for good instructional strategies, and (C) providing actionable suggestions for
eliciting more student reasoning. We recruit expert math teachers to evaluate
the zero-shot performance of ChatGPT on each of these tasks for elementary math
classroom transcripts. Our results reveal that ChatGPT generates responses that
are relevant to improving instruction, but they are often not novel or
insightful. For example, 82% of the model's suggestions point to places in the
transcript where the teacher is already implementing that suggestion. Our work
highlights the challenges of producing insightful, novel and truthful feedback
for teachers while paving the way for future research to address these
obstacles and improve the capacity of generative AI to coach teachers.

本研究探讨是否生成式人工智能可以作为自动化教练辅助专家反馈以成为对教师训练的一种有效补充。在这个过程中，针对小学数学课堂记录文本，我们提出了生成式人工智能的三种教师培训任务，并邀请数学领域专家评估了 ChatGPT 模型在每种任务上的表现。研究显示，生成式 AI 提供的反馈意见虽然往往不是新颖或有洞察力的，但它们仍然与改善教学相关。

ChatGPT 是否是一位出色的教师辅导？通过测量零 - shot 性能，为课堂教学评分和提供可操作洞见

Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For  Scoring and Providing Actionable Insights on Classroom Instruction

It has been commonly observed that a teacher model with superior performance
does not necessarily result in a stronger student, highlighting a discrepancy
between current teacher training practices and effective knowledge transfer. In
order to enhance the guidance of the teacher training process, we introduce the
concept of distillation influence to determine the impact of distillation from
each training sample on the student's generalization ability. In this paper, we
propose Learning Good Teacher Matters (LGTM), an efficient training technique
for incorporating distillation influence into the teacher's learning process.
By prioritizing samples that are likely to enhance the student's generalization
ability, our LGTM outperforms 10 common knowledge distillation baselines on 6
text classification tasks in the GLUE benchmark.

本文提出了一种名为 LGTM 的有效教师培训技术，该技术通过引入蒸馏影响概念来确定每个训练样本对学生泛化能力的影响，通过优先选择可能增强学生泛化能力的样本，在 GLUE 基准测试数据集的 6 项文本分类任务中，LGTM 比其他 10 种常见的知识蒸馏基线表现更好。