Reinforcement learning (RL) has shown promise as a tool for engineering safe,
ethical, or legal behaviour in autonomous agents. Its use typically relies on
assigning punishments to state-action pairs that constitute unsafe or unethical
choices. Despite this assignment being a crucial step in this approach,
however, there has been limited discussion on generalizing the process of
selecting punishments and deciding where to apply them. In this paper, we adopt
an approach that leverages an existing framework -- the normative supervisor of
(Neufeld et al., 2021) -- during training. This normative supervisor is used to
dynamically translate states and the applicable normative system into
defeasible deontic logic theories, feed these theories to a theorem prover, and
use the conclusions derived to decide whether or not to assign a punishment to
the agent. We use multi-objective RL (MORL) to balance the ethical objective of
avoiding violations with a non-ethical objective; we will demonstrate that our
approach works for a multiplicity of MORL techniques, and show that it is
effective regardless of the magnitude of the punishment we assign.

本文展示了一种使用利用现有的 normative supervisor 框架的方法，通过 MORL 技术以平衡非道德目标和避免违规的道德目标，在自主代理中实现安全、道德或合法行为，且不受惩罚强度大小的影响。