Conventional knowledge distillation (KD) methods require access to the
internal information of teachers, e.g., logits. However, such information may
not always be accessible for large pre-trained language models (PLMs). In this
work, we focus on decision-based KD for PLMs, where only teacher decisions
(i.e., top-1 labels) are accessible. Considering the information gap between
logits and decisions, we propose a novel method to estimate logits from the
decision distributions. Specifically, decision distributions can be both
derived as a function of logits theoretically and estimated with test-time data
augmentation empirically. By combining the theoretical and empirical
estimations of the decision distributions together, the estimation of logits
can be successfully reduced to a simple root-finding problem. Extensive
experiments show that our method significantly outperforms strong baselines on
both natural language understanding and machine reading comprehension datasets.

研究新的决策 KD 技术，从决策分布中估算出 logits，同时将理论和实验方法相结合，提高了自然语言理解和机器阅读理解性能。