Gorman and Bedrick (2019) argued for using random splits rather than standard
splits in NLP experiments. We argue that random splits, like standard splits,
lead to overly optimistic performance estimates. We can also split data in
biased or adversarial ways, e.g., training on short sentences and evaluating on
long ones. Biased sampling has been used in domain adaptation to simulate
real-world drift; this is known as the covariate shift assumption. In NLP,
however, even worst-case splits, maximizing bias, often under-estimate the
error observed on new samples of in-domain data, i.e., the data that models
should minimally generalize to at test time. This invalidates the covariate
shift assumption. Instead of using multiple random splits, future benchmarks
should ideally include multiple, independent test sets instead; if infeasible,
we argue that multiple biased splits leads to more realistic performance
estimates than multiple random splits.

研究指出，NLP 领域的实验应当使用多重的、独立的测试集进行评估，以获得更真实的性能评估；当不可能使用多个测试集时，使用多个有偏差的划分方式可得到更接近真实的性能评估。