Early stopping based on the validation set performance is a popular approach
to find the right balance between under- and overfitting in the context of
supervised learning. However, in reinforcement learning, even for supervised
sub-problems such as world model learning, early stopping is not applicable as
the dataset is continually evolving. As a solution, we propose a new general
method that dynamically adjusts the update to data (UTD) ratio during training
based on under- and overfitting detection on a small subset of the continuously
collected experience not used for training. We apply our method to DreamerV2, a
state-of-the-art model-based reinforcement learning algorithm, and evaluate it
on the DeepMind Control Suite and the Atari $100$k benchmark. The results
demonstrate that one can better balance under- and overestimation by adjusting
the UTD ratio with our approach compared to the default setting in DreamerV2
and that it is competitive with an extensive hyperparameter search which is not
feasible for many applications. Our method eliminates the need to set the UTD
hyperparameter by hand and even leads to a higher robustness with regard to
other learning-related hyperparameters further reducing the amount of necessary
tuning.

在强化学习等连续数据环境中，通过在未使用的一小部分数据上检测欠拟合和过拟合从而动态调整数据更新比率的新方法，比默认设置比更好的平衡欠拟合和过度估计，消除了手动设置超参数的需要，并使模型具有更高的健壮性，同时减少了必要的调试量。