Empirical design in reinforcement learning is no small task. Running good
experiments requires attention to detail and at times significant computational
resources. While compute resources available per dollar have continued to grow
rapidly, so have the scale of typical experiments in reinforcement learning. It
is now common to benchmark agents with millions of parameters against dozens of
tasks, each using the equivalent of 30 days of experience. The scale of these
experiments often conflict with the need for proper statistical evidence,
especially when comparing algorithms. Recent studies have highlighted how
popular algorithms are sensitive to hyper-parameter settings and implementation
details, and that common empirical practice leads to weak statistical evidence
(Machado et al., 2018; Henderson et al., 2018). Here we take this one step
further.
This manuscript represents both a call to action, and a comprehensive
resource for how to do good experiments in reinforcement learning. In
particular, we cover: the statistical assumptions underlying common performance
measures, how to properly characterize performance variation and stability,
hypothesis testing, special considerations for comparing multiple agents,
baseline and illustrative example construction, and how to deal with
hyper-parameters and experimenter bias. Throughout we highlight common mistakes
found in the literature and the statistical consequences of those in example
experiments. The objective of this document is to provide answers on how we can
use our unprecedented compute to do good science in reinforcement learning, as
well as stay alert to potential pitfalls in our empirical design.

本文旨在提出在强化学习中进行良好实验的方法，并强调常见错误和潜在统计结果，覆盖了如何对性能进行妥善表征、假设检验、比较多个代理、基准和说明性例子的构建、如何处理超参数和实验偏差等等，旨在通过充分利用计算资源来进行良好的实证研究。

强化学习中的经验设计

Empirical Design in Reinforcement Learning

Many man-made and natural phenomena, including the intensity of earthquakes,
population of cities and size of international wars, are believed to follow
power-law distributions. The accurate identification of power-law patterns has
significant consequences for correctly understanding and modeling complex
systems. However, statistical evidence for or against the power-law hypothesis
is complicated by large fluctuations in the empirical distribution's tail, and
these are worsened when information is lost from binning the data. We adapt the
statistically principled framework for testing the power-law hypothesis,
developed by Clauset, Shalizi and Newman, to the case of binned data. This
approach includes maximum-likelihood fitting, a hypothesis test based on the
Kolmogorov--Smirnov goodness-of-fit statistic and likelihood ratio tests for
comparing against alternative explanations. We evaluate the effectiveness of
these methods on synthetic binned data with known structure, quantify the loss
of statistical power due to binning, and apply the methods to twelve real-world
binned data sets with heavy-tailed patterns.

本研究采用最大似然拟合、基于 Kolmogorov-Smirnov 拟合度量的假设检验和可能性比检验等方法，针对实际的具有重尾性状的 12 个不同数据集进行分析，以评估用于有所不同的描述方式下的第三方统计学中有关幂律分布的假设检验的效果，并量化了数据划分所导致的统计功率损失。