Multi-objective reinforcement learning (MORL) algorithms tackle sequential
decision problems where agents may have different preferences over (possibly
conflicting) reward functions. Such algorithms often learn a set of policies
(each optimized for a particular agent preference) that can later be used to
solve problems with novel preferences. We introduce a novel algorithm that uses
Generalized Policy Improvement (GPI) to define principled, formally-derived
prioritization schemes that improve sample-efficient learning. They implement
active-learning strategies by which the agent can (i) identify the most
promising preferences/objectives to train on at each moment, to more rapidly
solve a given MORL problem; and (ii) identify which previous experiences are
most relevant when learning a policy for a particular agent preference, via a
novel Dyna-style MORL method. We prove our algorithm is guaranteed to always
converge to an optimal solution in a finite number of steps, or an
$\epsilon$-optimal solution (for a bounded $\epsilon$) if the agent is limited
and can only identify possibly sub-optimal policies. We also prove that our
method monotonically improves the quality of its partial solutions while
learning. Finally, we introduce a bound that characterizes the maximum utility
loss (with respect to the optimal solution) incurred by the partial solutions
computed by our method throughout learning. We empirically show that our method
outperforms state-of-the-art MORL algorithms in challenging multi-objective
tasks, both with discrete and continuous state and action spaces.

介绍了一种新的多目标强化学习算法，使用广义策略提升来定义优先级，实现了积极的学习策略，在学习中获得更高效的样本，通过使用 Dyna 样式的 MORL 方法识别先前经验最相关的特定代理偏好的策略以提高学习效率，并证明了算法始终收敛于一个有限步数的最优解，同时单调地提高其部分解决方案的质量。