Learning of preference models from human feedback has been central to recent
advances in artificial intelligence. Motivated by this progress, and the cost
of obtaining high-quality human annotations, we study the problem of data
collection for learning preference models. The key idea in our work is to
generalize optimal designs, a tool for computing efficient data logging
policies, to ranked lists. To show the generality of our ideas, we study both
absolute and relative feedback on items in the list. We design efficient
algorithms for both settings and analyze them. We prove that our preference
model estimators improve with more data and so does the ranking error under the
estimators. Finally, we experiment with several synthetic and real-world
datasets to show the statistical efficiency of our algorithms.

从人类反馈中学习偏好模型一直是人工智能领域最近进展的核心。本研究通过推广最优设计的概念，研究了用于学习偏好模型的数据收集问题，并提出了面向排名列表的有效算法，证明了模型估计器随更多数据而改善，估计器下的排名误差也随之减少，并在多个合成和真实数据集上进行实验以展示算法的统计效率。