In classic reinforcement learning algorithms, agents make decisions at
discrete and fixed time intervals. The physical duration between one decision
and the next becomes a critical hyperparameter. When this duration is too
short, the agent needs to make many decisions to achieve its goal, aggravating
the problem's difficulty. But when this duration is too long, the agent becomes
incapable of controlling the system. Physical systems, however, do not need a
constant control frequency. For learning agents, it is desirable to operate
with low frequency when possible and high frequency when necessary. We propose
a framework called Continuous-Time Continuous-Options (CTCO), where the agent
chooses options as sub-policies of variable durations. Such options are
time-continuous and can interact with the system at any desired frequency
providing a smooth change of actions. The empirical analysis shows that our
algorithm is competitive w.r.t. other time-abstraction techniques, such as
classic option learning and action repetition, and practically overcomes the
difficult choice of the decision frequency.

本研究提出一种名为 CTCO 的框架，使学习智能体通过选择变量持续时间的子策略来实现在可能的情况下以低频率运作，并在必要时以高频率运作，从而克服了决策频率选择的困难。