The goal of multi-objective reinforcement learning (MORL) is to learn
policies that simultaneously optimize multiple competing objectives. In
practice, an agent's preferences over the objectives may not be known apriori,
and hence, we require policies that can generalize to arbitrary preferences at
test time. In this work, we propose a new data-driven setup for offline MORL,
where we wish to learn a preference-agnostic policy agent using only a finite
dataset of offline demonstrations of other agents and their preferences. The
key contributions of this work are two-fold. First, we introduce D4MORL,
(D)atasets for MORL that are specifically designed for offline settings. It
contains 1.8 million annotated demonstrations obtained by rolling out reference
policies that optimize for randomly sampled preferences on 6 MuJoCo
environments with 2-3 objectives each. Second, we propose Pareto-Efficient
Decision Agents (PEDA), a family of offline MORL algorithms that builds and
extends Decision Transformers via a novel preference-and-return-conditioned
policy. Empirically, we show that PEDA closely approximates the behavioral
policy on the D4MORL benchmark and provides an excellent approximation of the
Pareto-front with appropriate conditioning, as measured by the hypervolume and
sparsity metrics.

本文提出了一种新的数据驱动离线 MORL 设置，介绍了专门针对离线设置的数据集 D4MORL，提出了一种基于 Pareto-Efficient Decision Agents 算法的决策代理，这种代理在行为策略上表现十分接近，在适当的情况下提供了 Pareto-front 的良好近似，可以通过超体积和稀疏度度量来衡量。