We consider a class of reinforcement-learning systems in which the agent
follows a behavior policy to explore a discrete state-action space to find an
optimal policy while adhering to some restriction on its behavior. Such
restriction may prevent the agent from visiting some state-action pairs,
possibly leading to the agent finding only a sub-optimal policy. To address
this problem we introduce the concept of constrained exploration with
optimality preservation, whereby the exploration behavior of the agent is
constrained to meet a specification while the optimality of the (original)
unconstrained learning process is preserved. We first establish a
feedback-control structure that models the dynamics of the unconstrained
learning process. We then extend this structure by adding a supervisor to
ensure that the behavior of the agent meets the specification, and establish
(for a class of reinforcement-learning problems with a known deterministic
environment) a necessary and sufficient condition under which optimality is
preserved. This work demonstrates the utility and the prospect of studying
reinforcement-learning problems in the context of the theories of
discrete-event systems, automata and formal languages.

在强化学习问题中引入概念的受限探索与最优保持，在满足某些约束时保持学习的最优性，通过引入监督器控制行为，建立了一个反馈控制结构来建模无约束学习过程的动态，为知道确定性环境的强化学习问题建立了必要条件和充分条件。