Existing multi-objective reinforcement learning (MORL) algorithms do not
account for objectives that arise from players with differing beliefs.
Concretely, consider two players with different beliefs and utility functions
who may cooperate to build a machine that takes actions on their behalf. A
representation is needed for how much the machine's policy will prioritize each
player's interests over time. Assuming the players have reached common
knowledge of their situation, this paper derives a recursion that any Pareto
optimal policy must satisfy. Two qualitative observations can be made from the
recursion: the machine must (1) use each player's own beliefs in evaluating how
well an action will serve that player's utility function, and (2) shift the
relative priority it assigns to each player's expected utilities over time, by
a factor proportional to how well that player's beliefs predict the machine's
inputs. Observation (2) represents a substantial divergence from na\"{i}ve
linear utility aggregation (as in Harsanyi's utilitarian theorem, and existing
MORL algorithms), which is shown here to be inadequate for Pareto optimal
sequential decision-making on behalf of players with different beliefs.

本文研究多目标强化学习算法在博弈中不同信念玩家的目标问题，提出了一种策略表述方式，利用递归形式来得出 Pareto 最优策略的特征，其中包括机器学习会利用玩家自身的信念来评估策略效果，并根据信念和机器输入条件的贴合程度，适当调整对每个玩家效用的相对优先级。