Preference-based reward learning is a popular technique for teaching robots
and autonomous systems how a human user wants them to perform a task. Previous
works have shown that actively synthesizing preference queries to maximize
information gain about the reward function parameters improves data efficiency.
The information gain criterion focuses on precisely identifying all parameters
of the reward function. This can potentially be wasteful as many parameters may
result in the same reward, and many rewards may result in the same behavior in
the downstream tasks. Instead, we show that it is possible to optimize for
learning the reward function up to a behavioral equivalence class, such as
inducing the same ranking over behaviors, distribution over choices, or other
related definitions of what makes two rewards similar. We introduce a tractable
framework that can capture such definitions of similarity. Our experiments in a
synthetic environment, an assistive robotics environment with domain transfer,
and a natural language processing problem with real datasets demonstrate the
superior performance of our querying method over the state-of-the-art
information gain method.

优化查询方法在学习奖励函数上的表现优于信息增益方法。