Experimental studies are a cornerstone of machine learning (ML) research. A
common, but often implicit, assumption is that the results of a study will
generalize beyond the study itself, e.g. to new data. That is, there is a high
probability that repeating the study under different conditions will yield
similar results. Despite the importance of the concept, the problem of
measuring generalizability remains open. This is probably due to the lack of a
mathematical formalization of experimental studies. In this paper, we propose
such a formalization and develop a quantifiable notion of generalizability.
This notion allows to explore the generalizability of existing studies and to
estimate the number of experiments needed to achieve the generalizability of
new studies. To demonstrate its usefulness, we apply it to two recently
published benchmarks to discern generalizable and non-generalizable results. We
also publish a Python module that allows our analysis to be repeated for other
experimental studies.

本文提出了实验研究的数学形式化方法，并发展了一个可量化的泛化概念，旨在探索现有研究的泛化性，并估计实现新研究泛化性所需的实验数量，为此我们应用两个最近发表的基准测试来区分可泛化和不可泛化的结果，并发布了一个 Python 模块，可以用于其他实验研究的分析。