Planning in real-world settings often entails addressing partial
observability while aligning with users' preferences. We present a novel
framework for expressing users' preferences about agent behavior in a partially
observable setting using parameterized belief-state query (BSQ) preferences in
the setting of goal-oriented partially observable Markov decision processes
(gPOMDPs). We present the first formal analysis of such preferences and prove
that while the expected value of a BSQ preference is not a convex function
w.r.t its parameters, it is piecewise constant and yields an implicit discrete
parameter search space that is finite for finite horizons. This theoretical
result leads to novel algorithms that optimize gPOMDP agent behavior while
guaranteeing user preference compliance. Theoretical analysis proves that our
algorithms converge to the optimal preference-compliant behavior in the limit.
Empirical results show that BSQ preferences provide a computationally feasible
approach for planning with preferences in partially observable settings.

利用参数化信念查询（BSQ）偏好在部分可观测环境中表达用户对代理行为的偏好的新框架，在目标导向的部分可观测马尔可夫决策过程（gPOMDPs）的设定中进行介绍。我们首次对这种偏好进行了形式化分析，并证明 BSQ 偏好的期望值虽然不是关于其参数的凸函数，但具有分段常数特性，并产生一个对于有限时间限制而言有限的离散参数搜索空间。这个理论结果导致了新的算法，可以在优化 gPOMDP 代理行为的同时保证用户偏好的一致性。理论分析证明了我们的算法在极限情况下收敛于最优的偏好一致行为。实证结果表明，在部分可观测环境中，BSQ 偏好为基于偏好的规划提供了一种计算上可行的方法。