Information access systems, such as search engines, recommender systems, and
conversational assistants, have become integral to our daily lives as they help
us satisfy our information needs. However, evaluating the effectiveness of
these systems presents a long-standing and complex scientific challenge. This
challenge is rooted in the difficulty of assessing a system's overall
effectiveness in assisting users to complete tasks through interactive support,
and further exacerbated by the substantial variation in user behaviour and
preferences. To address this challenge, user simulation emerges as a promising
solution.
This book focuses on providing a thorough understanding of user simulation
techniques designed specifically for evaluation purposes. We begin with a
background of information access system evaluation and explore the diverse
applications of user simulation. Subsequently, we systematically review the
major research progress in user simulation, covering both general frameworks
for designing user simulators, utilizing user simulation for evaluation, and
specific models and algorithms for simulating user interactions with search
engines, recommender systems, and conversational assistants. Realizing that
user simulation is an interdisciplinary research topic, whenever possible, we
attempt to establish connections with related fields, including machine
learning, dialogue systems, user modeling, and economics. We end the book with
a detailed discussion of important future research directions, many of which
extend beyond the evaluation of information access systems and are expected to
have broader impact on how to evaluate interactive intelligent systems in
general.

本书系统地回顾了用户模拟的研究进展，包括为设计用户模拟器制定的一般框架、利用用户模拟进行评估的模型和算法，以及用于搜索引擎、推荐系统和对话助手的用户交互模拟，并讨论了重要的未来研究方向。

评估信息获取系统的用户模拟

User Simulation for Evaluating Information Access Systems

Although information access systems have long supported people in
accomplishing a wide range of tasks, we propose broadening the scope of users
of information access systems to include task-driven machines, such as machine
learning models. In this way, the core principles of indexing, representation,
retrieval, and ranking can be applied and extended to substantially improve
model generalization, scalability, robustness, and interpretability. We
describe a generic retrieval-enhanced machine learning (REML) framework, which
includes a number of existing models as special cases. REML challenges
information retrieval conventions, presenting opportunities for novel advances
in core areas, including optimization. The REML research agenda lays a
foundation for a new style of information access research and paves a path
towards advancing machine learning and artificial intelligence.

这篇论文提出了一种通用的增强检索的机器学习框架，可以将信息检索的核心原理应用和扩展到任务驱动的机器，从而实现模型的泛化、可扩展性、鲁棒性和可解释性的显著提高。