Sequential decision-making is desired to align with human intents and exhibit versatility across various tasks. Previous methods formulate it as a conditional generation process, utilizing return-conditioned diffusion models to directly model trajectory distributions. Nevertheless, the return-conditioned paradigm relies on pre-defined reward functions, facing challenges when applied in multi-task settings characterized by varying reward functions (versatility) and showing limited controllability concerning human preferences (alignment). In this work, we adopt multi-task preferences as a unified condition for both single- and multi-task decision-making, and propose preference representations aligned with preference labels. The learned representations are used to guide the conditional generation process of diffusion models, and we introduce an auxiliary objective to maximize the mutual information between representations and corresponding generated trajectories, improving alignment between trajectories and preferences. Extensive experiments in D4RL and Meta-World demonstrate that our method presents favorable performance in single- and multi-task scenarios, and exhibits superior alignment with preferences.

在这项工作中，我们采用多任务偏好作为统一条件，既适用于单任务又适用于多任务决策，提出了与偏好标签对齐的偏好表示方法。通过引入一个辅助目标来最大化表示和生成的轨迹之间的互信息，改善了轨迹和偏好之间的对齐。在D4RL和Meta-World的大量实验证明，我们的方法在单任务和多任务场景中表现出良好的性能，并展现出与偏好的优越对齐性。

多任务偏好对齐的正则化条件扩散模型