We present a large-scale empirical study of how choices of configuration
parameters affect performance in knowledge distillation (KD). An example of
such a KD parameter is the measure of distance between the predictions of the
teacher and the student, common choices for which include the mean squared
error (MSE) and the KL-divergence. Although scattered efforts have been made to
understand the differences between such options, the KD literature still lacks
a systematic study on their general effect on student performance. We take an
empirical approach to this question in this paper, seeking to find out the
extent to which such choices influence student performance across 13 datasets
from 4 NLP tasks and 3 student sizes. We quantify the cost of making
sub-optimal choices and identify a single configuration that performs well
across the board.

本文通过大规模实证研究，探讨配置参数选择对知识蒸馏（KD）性能的影响，以及如何在 13 个数据集、4 个 NLP 任务和 3 种学生规模下对学生性能的影响进行评估，最终确定一种在各方面表现良好的配置。

知识蒸馏中参数选择的实证研究

An Empirical Investigation into the Effect of Parameter Choices in  Knowledge Distillation

A large number of real-world optimization problems can be formulated as Mixed
Integer Linear Programs (MILP). MILP solvers expose numerous configuration
parameters to control their internal algorithms. Solutions, and their
associated costs or runtimes, are significantly affected by the choice of the
configuration parameters, even when problem instances have the same number of
decision variables and constraints. On one hand, using the default solver
configuration leads to suboptimal solutions. On the other hand, searching and
evaluating a large number of configurations for every problem instance is
time-consuming and, in some cases, infeasible. In this study, we aim to predict
configuration parameters for unseen problem instances that yield lower-cost
solutions without the time overhead of searching-and-evaluating configurations
at the solving time. Toward that goal, we first investigate the cost
correlation of MILP problem instances that come from the same distribution when
solved using different configurations. We show that instances that have similar
costs using one solver configuration also have similar costs using another
solver configuration in the same runtime environment. After that, we present a
methodology based on Deep Metric Learning to learn MILP similarities that
correlate with their final solutions' costs. At inference time, given a new
problem instance, it is first projected into the learned metric space using the
trained model, and configuration parameters are instantly predicted using
previously-explored configurations from the nearest neighbor instance in the
learned embedding space. Empirical results on real-world problem benchmarks
show that our method predicts configuration parameters that improve solutions'
costs by up to 38% compared to existing approaches.

本研究旨在使用深度度量学习方法，预测出 MILP 问题的配置参数，以获得更低成本的解决方案。经过实证结果表明，相比于现有方法，本方法能够提高解决方案的成本高达 38%。

通过学习问题相似性自动配置 MILP 求解器

Automatic MILP Solver Configuration By Learning Problem Similarities

Distributed file systems are widely used nowadays, yet using their default
configurations is often not optimal. At the same time, tuning configuration
parameters is typically challenging and time-consuming. It demands expertise
and tuning operations can also be expensive. This is especially the case for
static parameters, where changes take effect only after a restart of the system
or workloads. We propose a novel approach, Magpie, which utilizes deep
reinforcement learning to tune static parameters by strategically exploring and
exploiting configuration parameter spaces. To boost the tuning of the static
parameters, our method employs both server and client metrics of distributed
file systems to understand the relationship between static parameters and
performance. Our empirical evaluation results show that Magpie can noticeably
improve the performance of the distributed file system Lustre, where our
approach on average achieves 91.8% throughput gains against default
configuration after tuning towards single performance indicator optimization,
while it reaches 39.7% more throughput gains against the baseline.

使用深度增强学习优化分布式文件系统的性能，提高系统吞吐量最高可达 91.8%，相对于基线提升了 39.7%。