Robust reinforcement learning is essential for deploying reinforcement
learning algorithms in real-world scenarios where environmental uncertainty
predominates. Traditional robust reinforcement learning often depends on
rectangularity assumptions, where adverse probability measures of outcome
states are assumed to be independent across different states and actions. This
assumption, rarely fulfilled in practice, leads to overly conservative
policies. To address this problem, we introduce a new time-constrained robust
MDP (TC-RMDP) formulation that considers multifactorial, correlated, and
time-dependent disturbances, thus more accurately reflecting real-world
dynamics. This formulation goes beyond the conventional rectangularity
paradigm, offering new perspectives and expanding the analytical framework for
robust RL. We propose three distinct algorithms, each using varying levels of
environmental information, and evaluate them extensively on continuous control
benchmarks. Our results demonstrate that these algorithms yield an efficient
tradeoff between performance and robustness, outperforming traditional deep
robust RL methods in time-constrained environments while preserving robustness
in classical benchmarks. This study revisits the prevailing assumptions in
robust RL and opens new avenues for developing more practical and realistic RL
applications.

通过引入新的时间约束鲁棒马尔科夫决策过程（TC-RMDP）表达方式，考虑到多因素、相关性和时变干扰，该研究重新审视了鲁棒强化学习中的传统假设，为发展更实际、更真实的强化学习应用开辟了新的路径，同时在时间受限环境下，在保持鲁棒性的同时，取得了性能和鲁棒性之间的高效平衡。