Empirical design in reinforcement learning is no small task. Running good
experiments requires attention to detail and at times significant computational
resources. While compute resources available per dollar have continued to grow
rapidly, so have the scale of typical experiments in reinforcement learning. It
is now common to benchmark agents with millions of parameters against dozens of
tasks, each using the equivalent of 30 days of experience. The scale of these
experiments often conflict with the need for proper statistical evidence,
especially when comparing algorithms. Recent studies have highlighted how
popular algorithms are sensitive to hyper-parameter settings and implementation
details, and that common empirical practice leads to weak statistical evidence
(Machado et al., 2018; Henderson et al., 2018). Here we take this one step
further.
This manuscript represents both a call to action, and a comprehensive
resource for how to do good experiments in reinforcement learning. In
particular, we cover: the statistical assumptions underlying common performance
measures, how to properly characterize performance variation and stability,
hypothesis testing, special considerations for comparing multiple agents,
baseline and illustrative example construction, and how to deal with
hyper-parameters and experimenter bias. Throughout we highlight common mistakes
found in the literature and the statistical consequences of those in example
experiments. The objective of this document is to provide answers on how we can
use our unprecedented compute to do good science in reinforcement learning, as
well as stay alert to potential pitfalls in our empirical design.

本文旨在提出在强化学习中进行良好实验的方法，并强调常见错误和潜在统计结果，覆盖了如何对性能进行妥善表征、假设检验、比较多个代理、基准和说明性例子的构建、如何处理超参数和实验偏差等等，旨在通过充分利用计算资源来进行良好的实证研究。