Delayed Markov decision processes fulfill the Markov property by augmenting
the state space of agents with a finite time window of recently committed
actions. In reliance with these state augmentations, delay-resolved
reinforcement learning algorithms train policies to learn optimal interactions
with environments featured with observation or action delays. Although such
methods can directly be trained on the real robots, due to sample inefficiency,
limited resources or safety constraints, a common approach is to transfer
models trained in simulation to the physical robot. However, robotic
simulations rely on approximated models of the physical systems, which hinders
the sim2real transfer. In this work, we consider various uncertainties in the
modelling of the robot's dynamics as unknown intrinsic disturbances applied on
the system input. We introduce a disturbance-augmented Markov decision process
in delayed settings as a novel representation to incorporate disturbance
estimation in training on-policy reinforcement learning algorithms. The
proposed method is validated across several metrics on learning a robotic
reaching task and compared with disturbance-unaware baselines. The results show
that the disturbance-augmented models can achieve higher stabilization and
robustness in the control response, which in turn improves the prospects of
successful sim2real transfer.

本文研究了引入扰动估计的延迟条件下，通过扩展状态空间的方式应用马尔可夫决策过程的新方法，可以提高机器人控制的稳定性和鲁棒性。