Dynamic benchmarks interweave model fitting and data collection in an attempt
to mitigate the limitations of static benchmarks. In contrast to an extensive
theoretical and empirical study of the static setting, the dynamic counterpart
lags behind due to limited empirical studies and no apparent theoretical
foundation to date. Responding to this deficit, we initiate a theoretical study
of dynamic benchmarking. We examine two realizations, one capturing current
practice and the other modeling more complex settings. In the first model,
where data collection and model fitting alternate sequentially, we prove that
model performance improves initially but can stall after only three rounds.
Label noise arising from, for instance, annotator disagreement leads to even
stronger negative results. Our second model generalizes the first to the case
where data collection and model fitting have a hierarchical dependency
structure. We show that this design guarantees strictly more progress than the
first, albeit at a significant increase in complexity. We support our
theoretical analysis by simulating dynamic benchmarks on two popular datasets.
These results illuminate the benefits and practical limitations of dynamic
benchmarking, providing both a theoretical foundation and a causal explanation
for observed bottlenecks in empirical work.

本研究理论分析了动态基准测试的两种实现方式，第一种模型中，模型性能最初会有所提高，但只会在三轮后停滞，而第二种模型则保证了比第一种模型更多的进展，但复杂度更高，并通过模拟动态基准测试的结果来验证了理论分析，为动态基准测试提供了理论和实践上的支持。