Recommender systems operate in an inherently dynamical setting. Past
recommendations influence future behavior, including which data points are
observed and how user preferences change. However, experimenting in production
systems with real user dynamics is often infeasible, and existing
simulation-based approaches have limited scale. As a result, many
state-of-the-art algorithms are designed to solve supervised learning problems,
and progress is judged only by offline metrics. In this work we investigate the
extent to which offline metrics predict online performance by evaluating eleven
recommenders across six controlled simulated environments. We observe that
offline metrics are correlated with online performance over a range of
environments. However, improvements in offline metrics lead to diminishing
returns in online performance. Furthermore, we observe that the ranking of
recommenders varies depending on the amount of initial offline data available.
We study the impact of adding exploration strategies, and observe that their
effectiveness, when compared to greedy recommendation, is highly dependent on
the recommendation algorithm. We provide the environments and recommenders
described in this paper as Reclab: an extensible ready-to-use simulation
framework at this https URL

通过在六个受控的模拟环境中对 11 个推荐系统进行评估，研究离线度量是否能够预测在线性能，在观察到离线度量在一定范围内与在线性能相关的同时，还发现离线性能的提高会导致在线性能的收益递减，同时观察到推荐系统的排名取决于初始离线数据量以及探索策略的添加策略的有效性高度依赖推荐算法。