Data-driven science is an emerging paradigm where scientific discoveries
depend on the execution of computational AI models against rich,
discipline-specific datasets. With modern machine learning frameworks, anyone
can develop and execute computational models that reveal concepts hidden in the
data that could enable scientific applications. For important and widely used
datasets, computing the performance of every computational model that can run
against a dataset is cost prohibitive in terms of cloud resources. Benchmarking
approaches used in practice use representative datasets to infer performance
without actually executing models. While practicable, these approaches limit
extensive dataset profiling to a few datasets and introduce bias that favors
models suited for representative datasets. As a result, each dataset's unique
characteristics are left unexplored and subpar models are selected based on
inference from generalized datasets. This necessitates a new paradigm that
introduces dataset profiling into the model selection process. To demonstrate
the need for dataset-specific profiling, we answer two questions:(1) Can
scientific datasets significantly permute the rank order of computational
models compared to widely used representative datasets? (2) If so, could
lightweight model execution improve benchmarking accuracy? Taken together, the
answers to these questions lay the foundation for a new dataset-aware
benchmarking paradigm.

数据驱动科学是一种新兴的范例，其中科学发现取决于针对具体学科的丰富数据集执行计算 AI 模型。通过现代机器学习框架，任何人都可以开发和执行计算模型，揭示隐藏在数据中的概念，从而可能支持科学应用。然而，在实际应用中，收集并计算每个可以运行的计算模型的性能代价昂贵。由于使用代表性数据集来推断性能的基准测试方法具有局限性，每个数据集都具有独特的特征，这需要引入数据集配置文件来作为模型选择过程的一部分，以选择最佳模型进行优化。