Strong empirical evidence that one machine-learning algorithm A outperforms
another one B ideally calls for multiple trials optimizing the learning
pipeline over sources of variation such as data sampling, data augmentation,
parameter initialization, and hyperparameters choices. This is prohibitively
expensive, and corners are cut to reach conclusions. We model the whole
benchmarking process, revealing that variance due to data sampling, parameter
initialization and hyperparameter choice impact markedly the results. We
analyze the predominant comparison methods used today in the light of this
variance. We show a counter-intuitive result that adding more sources of
variation to an imperfect estimator approaches better the ideal estimator at a
51 times reduction in compute cost. Building on these results, we study the
error rate of detecting improvements, on five different deep-learning
tasks/architectures. This study leads us to propose recommendations for
performance comparisons.

通过模拟对比机器学习算法的整个基准测试过程，我们发现数据采样、参数初始化和超参数选择对结果的影响显著。进一步分析今天所用的主要比较方法，我们提出一种反直觉的结果，即在不增加计算成本的情况下，将更多的变化源添加到不完美的估计器中可以接近更理想的估计器。通过五个不同的深度学习任务和架构，分析改进检测错误率，提出了性能比较的建议。