It is believed that in knowledge distillation (KD), the role of the teacher is to provide an estimate for the unknown Bayes conditional probability distribution (BCPD) to be used in the student training process. Conventionally, this estimate is obtained by training the teacher using maximum log-likelihood (MLL) method. To improve this estimate for KD, in this paper we introduce the concept of conditional mutual information (CMI) into the estimation of BCPD and propose a novel estimator called the maximum CMI (MCMI) method. Specifically, in MCMI estimation, both the log-likelihood and CMI of the teacher are simultaneously maximized when the teacher is trained. Through Eigen-CAM, it is further shown that maximizing the teacher's CMI value allows the teacher to capture more contextual information in an image cluster. Via conducting a thorough set of experiments, we show that by employing a teacher trained via MCMI estimation rather than one trained via MLL estimation in various state-of-the-art KD frameworks, the student's classification accuracy consistently increases, with the gain of up to 3.32\%. This suggests that the teacher's BCPD estimate provided by MCMI method is more accurate than that provided by MLL method. In addition, we show that such improvements in the student's accuracy are more drastic in zero-shot and few-shot settings. Notably, the student's accuracy increases with the gain of up to 5.72\% when 5\% of the training samples are available to the student (few-shot), and increases from 0\% to as high as 84\% for an omitted class (zero-shot). The code is available at \url{https://github.com/iclr2024mcmi/ICLRMCMI}.

通过引入条件互信息的概念以提高知识蒸馏中教师对未知贝叶斯条件概率分布的估计，本文提出了一种称为最大条件互信息方法的新估计器。通过对教师进行最大条件互信息估计，能够捕捉到图像聚类中更多的上下文信息，从而在各种先进的知识蒸馏框架中使用通过最大条件互信息估计而不是最大对数似然估计的教师，在学生的分类准确率上获得持续提高，增益高达3.32％。此外，研究还表明，这种提高在零样本和少样本情况下更为显著，当学生只有5％的训练样本可用（少样本）时，准确率提高了5.72％，在省略类别的情况下（零样本），准确率从0％增加到高达84％。

基于条件互信息的贝叶斯条件分布估计用于知识蒸馏