In this paper, we use concepts from supervisory control theory of discrete
event systems to propose a method to learn optimal control policies for a
finite-state Markov Decision Process (MDP) in which (only) certain sequences of
actions are deemed unsafe (respectively safe). We assume that the set of action
sequences that are deemed unsafe and/or safe are given in terms of a
finite-state automaton; and propose a supervisor that disables a subset of
actions at every state of the MDP so that the constraints on action sequence
are satisfied. Then we present a version of the Q-learning algorithm for
learning optimal policies in the presence of non-Markovian action-sequence and
state constraints, where we use the development of reward machines to handle
the state constraints. We illustrate the method using an example that captures
the utility of automata-based methods for non-Markovian state and action
specifications for reinforcement learning and show the results of simulations
in this setting.

本文运用离散事件系统监控控制理论的概念，提出一种方法用于在有限状态的马尔可夫决策过程中，学习最优控制策略，并利用奖励机器的发展来处理状态限制。通过给定一个例子来阐明其应用性并在此设置中展示了仿真结果。