Deep reinforcement learning algorithms that learn policies by trial-and-error must learn from limited amounts of data collected by actively interacting with the environment. While many prior works have shown that proper regularization techniques are crucial for enabling data-efficient RL, a general understanding of the bottlenecks in data-efficient RL has remained unclear. Consequently, it has been difficult to devise a universal technique that works well across all domains. In this paper, we attempt to understand the primary bottleneck in sample-efficient deep RL by examining several potential hypotheses such as non-stationarity, excessive action distribution shift, and overfitting. We perform thorough empirical analysis on state-based DeepMind control suite (DMC) tasks in a controlled and systematic way to show that high temporal-difference (TD) error on the validation set of transitions is the main culprit that severely affects the performance of deep RL algorithms, and prior methods that lead to good performance do in fact, control the validation TD error to be low. This observation gives us a robust principle for making deep RL efficient: we can hill-climb on the validation TD error by utilizing any form of regularization techniques from supervised learning. We show that a simple online model selection method that targets the validation TD error is effective across state-based DMC and Gym tasks.

本文通过对DeepMind控制套件中的任务进行控制和系统性分析，研究了数据高效RL的瓶颈，发现高TD错误是深度强化学习算法性能严重影响的主要罪魁祸首，因此，在任何形式的监督学习中，利用任何形式的正则化技术，找到验证TD误差的最低点是使深度RL高效的一个强有力的原则。一个简单的在线模型选择方法针对验证TD错误在基于状态的DMC和Gym任务中也是有效的。

高效深度强化学习需要控制过拟合