Offline reinforcement learning (RL purely from logged data) is an important
avenue for deploying RL techniques in real-world scenarios. However, existing
hyperparameter selection methods for offline RL break the offline assumption by
evaluating policies corresponding to each hyperparameter setting in the
environment. This online execution is often infeasible and hence undermines the
main aim of offline RL. Therefore, in this work, we focus on \textit{offline
hyperparameter selection}, i.e. methods for choosing the best policy from a set
of many policies trained using different hyperparameters, given only logged
data. Through large-scale empirical evaluation we show that: 1) offline RL
algorithms are not robust to hyperparameter choices, 2) factors such as the
offline RL algorithm and method for estimating Q values can have a big impact
on hyperparameter selection, and 3) when we control those factors carefully, we
can reliably rank policies across hyperparameter choices, and therefore choose
policies which are close to the best policy in the set. Overall, our results
present an optimistic view that offline hyperparameter selection is within
reach, even in challenging tasks with pixel observations, high dimensional
action spaces, and long horizon.

本篇论文研究了针对离线数据选择最佳 RL 策略的离线超参数选择方法，并通过大规模实证研究表明：1）离线 RL 算法对超参数的选择不具有健壮性，2）离线 RL 算法和 Q 值估计方法等因素对超参数选择具有较大影响，3）通过控制这些因素，我们可以可靠地对超参数选择进行排名，从而选择最接近设定中最佳策略的策略。