Tabular data is prevalent in real-world machine learning applications, and
new models for supervised learning of tabular data are frequently proposed.
Comparative studies assessing the performance of models typically consist of
model-centric evaluation setups with overly standardized data preprocessing.
This paper demonstrates that such model-centric evaluations are biased, as
real-world modeling pipelines often require dataset-specific preprocessing and
feature engineering. Therefore, we propose a data-centric evaluation framework.
We select 10 relevant datasets from Kaggle competitions and implement
expert-level preprocessing pipelines for each dataset. We conduct experiments
with different preprocessing pipelines and hyperparameter optimization (HPO)
regimes to quantify the impact of model selection, HPO, feature engineering,
and test-time adaptation. Our main findings are: 1. After dataset-specific
feature engineering, model rankings change considerably, performance
differences decrease, and the importance of model selection reduces. 2. Recent
models, despite their measurable progress, still significantly benefit from
manual feature engineering. This holds true for both tree-based models and
neural networks. 3. While tabular data is typically considered static, samples
are often collected over time, and adapting to distribution shifts can be
important even in supposedly static data. These insights suggest that research
efforts should be directed toward a data-centric perspective, acknowledging
that tabular data requires feature engineering and often exhibits temporal
characteristics.

通过从 Kaggle 竞赛中选择 10 个相关数据集，实现专家级预处理流程，并进行实验以量化模型选择、超参数优化、特征工程和测试时间适应的影响，我们的主要发现是：1. 数据集特定的特征工程后，模型排名发生了显著变化，性能差异减小，模型选择的重要性降低。2. 最新模型虽然取得了可衡量的进步，但仍然极大受益于手动特征工程，这对基于树的模型和神经网络都适用。3. 虽然通常认为表格数据是静态的，但样本经常随着时间的推移而收集，适应分布的变化即使在所谓的静态数据中也很重要。这些观点表明研究应关注数据为中心的视角，承认表格数据需要特征工程，并且通常具有时间特性。