An exciting and promising frontier for Deep Reinforcement Learning (DRL) is
its application to real-world robotic systems. While modern DRL approaches
achieved remarkable successes in many robotic scenarios (including mobile
robotics, surgical assistance, and autonomous driving) unpredictable and
non-stationary environments can pose critical challenges to such methods. These
features can significantly undermine fundamental requirements for a successful
training process, such as the Markovian properties of the transition model. To
address this challenge, we propose a new benchmarking environment for aquatic
navigation using recent advances in the integration between game engines and
DRL. In more detail, we show that our benchmarking environment is problematic
even for state-of-the-art DRL approaches that may struggle to generate reliable
policies in terms of generalization power and safety. Specifically, we focus on
PPO, one of the most widely accepted algorithms, and we propose advanced
training techniques (such as curriculum learning and learnable
hyperparameters). Our extensive empirical evaluation shows that a well-designed
combination of these ingredients can achieve promising results. Our simulation
environment and training baselines are freely available to facilitate further
research on this open problem and encourage collaboration in the field.

深度强化学习在应用于真实世界的机器人系统方面具有令人兴奋和有希望的前景。我们提出了一个基于最新的游戏引擎和深度强化学习整合的新的水下导航基准环境，以解决在不可预测和非稳态环境中训练过程中的挑战。我们使用先进的训练技术对这个基准环境进行了广泛的实证评估，取得了有希望的结果。