There has been significant progress in deep reinforcement learning (RL) in
recent years. Nevertheless, finding suitable hyperparameter configurations and
reward functions remains challenging even for experts, and performance heavily
relies on these design choices. Also, most RL research is conducted on known
benchmarks where knowledge about these choices already exists. However, novel
practical applications often pose complex tasks for which no prior knowledge
about good hyperparameters and reward functions is available, thus
necessitating their derivation from scratch. Prior work has examined
automatically tuning either hyperparameters or reward functions individually.
We demonstrate empirically that an RL algorithm's hyperparameter configurations
and reward function are often mutually dependent, meaning neither can be fully
optimised without appropriate values for the other. We then propose a
methodology for the combined optimisation of hyperparameters and the reward
function. Furthermore, we include a variance penalty as an optimisation
objective to improve the stability of learned policies. We conducted extensive
experiments using Proximal Policy Optimisation and Soft Actor-Critic on four
environments. Our results show that combined optimisation significantly
improves over baseline performance in half of the environments and achieves
competitive performance in the others, with only a minor increase in
computational costs. This suggests that combined optimisation should be best
practice.

深度强化学习在过去几年取得了显著进展，但是即使对于专家来说，找到适当的超参数配置和奖励函数仍然具有挑战性并且性能严重依赖于这些设计选择，因此我们提出了一种方法来同时优化超参数和奖励函数，实验证明综合优化可以显著改善性能，在一些环境中比基准性能好，并在其他环境中取得了竞争性表现，只有少量计算成本的增加。