Conventional knowledge distillation (KD) methods require access to the
internal information of teachers, e.g., logits. However, such information may
not always be accessible for large pre-trained language models (PLMs). In this
work, we focus on decision-based KD for PLMs, where only teacher decisions
(i.e., top-1 labels) are accessible. Considering the information gap between
logits and decisions, we propose a novel method to estimate logits from the
decision distributions. Specifically, decision distributions can be both
derived as a function of logits theoretically and estimated with test-time data
augmentation empirically. By combining the theoretical and empirical
estimations of the decision distributions together, the estimation of logits
can be successfully reduced to a simple root-finding problem. Extensive
experiments show that our method significantly outperforms strong baselines on
both natural language understanding and machine reading comprehension datasets.

研究新的决策 KD 技术，从决策分布中估算出 logits，同时将理论和实验方法相结合，提高了自然语言理解和机器阅读理解性能。

预训练语言模型中的基于决策的知识蒸馏中决策和逻辑之间的桥梁

Bridging the Gap between Decision and Logits in Decision-based Knowledge  Distillation for Pre-trained Language Models

This paper presents an effective and general data augmentation framework for
medical image segmentation. We adopt a computationally efficient and
data-efficient gradient-based meta-learning scheme to explicitly align the
distribution of training and validation data which is used as a proxy for
unseen test data. We improve the current data augmentation strategies with two
core designs. First, we learn class-specific training-time data augmentation
(TRA) effectively increasing the heterogeneity within the training subsets and
tackling the class imbalance common in segmentation. Second, we jointly
optimize TRA and test-time data augmentation (TEA), which are closely connected
as both aim to align the training and test data distribution but were so far
considered separately in previous works. We demonstrate the effectiveness of
our method on four medical image segmentation tasks across different scenarios
with two state-of-the-art segmentation models, DeepMedic and nnU-Net. Extensive
experimentation shows that the proposed data augmentation framework can
significantly and consistently improve the segmentation performance when
compared to existing solutions. Code is publicly available.

本文提出了一种针对医学图像分割的高效通用数据增强框架，采用了计算效率高且数据效率高的基于梯度的元学习方案，通过使用用于未见测试数据的代理验证数据明确对齐训练和验证数据的分布以改进增广策略。本文通过两个核心设计改进了当前数据增广策略，即在训练时有效学习类别特定的数据增强并联合优化训练时和测试时数据增强，实验证明该方法针对医学图像分割存在的分类不平衡问题，可显著且一致地提高分割效果。

在分割中联合优化类别特定的训练和测试时数据增强

Joint Optimization of Class-Specific Training- and Test-Time Data  Augmentation in Segmentation

Test-time data augmentation$-$averaging the predictions of a machine learning
model across multiple augmented samples of data$-$is a widely used technique
that improves the predictive performance. While many advanced learnable data
augmentation techniques have emerged in recent years, they are focused on the
training phase. Such techniques are not necessarily optimal for test-time
augmentation and can be outperformed by a policy consisting of simple crops and
flips. The primary goal of this paper is to demonstrate that test-time
augmentation policies can be successfully learned too. We introduce greedy
policy search (GPS), a simple but high-performing method for learning a policy
of test-time augmentation. We demonstrate that augmentation policies learned
with GPS achieve superior predictive performance on image classification
problems, provide better in-domain uncertainty estimation, and improve the
robustness to domain shift.

本文介绍了一种名为贪心策略搜索（GPS）的高性能方法，可用于学习数据增强的策略，以提高机器学习模型的预测性能、领域内不确定性估计以及提高对领域转移的鲁棒性分析。