We investigate model assessment and selection in a changing environment, by
synthesizing datasets from both the current time period and historical epochs.
To tackle unknown and potentially arbitrary temporal distribution shift, we
develop an adaptive rolling window approach to estimate the generalization
error of a given model. This strategy also facilitates the comparison between
any two candidate models by estimating the difference of their generalization
errors. We further integrate pairwise comparisons into a single-elimination
tournament, achieving near-optimal model selection from a collection of
candidates. Theoretical analyses and numerical experiments demonstrate the
adaptivity of our proposed methods to the non-stationarity in data.

我们通过综合当前时期和历史时期的数据集，研究模型评估和选择在变化环境中的情况。为了应对未知且可能随机的时间分布转移，我们开发了一种自适应滚动窗口方法来估计给定模型的泛化误差。该策略还通过估计两个候选模型的泛化误差差异来方便比较。我们进一步将两两比较整合到单一淘汰赛中，从候选模型集合中实现接近最优的模型选择。理论分析和数值实验证明了我们提出的方法对数据非稳定性的适应性。