Tabular data from different tables exhibit significant diversity due to
varied definitions and types of features, as well as complex inter-feature and
feature-target relationships. Cross-dataset pretraining, which learns reusable
patterns from upstream data to support downstream tasks, have shown notable
success in various fields. Yet, when applied to tabular data prediction, this
paradigm faces challenges due to the limited reusable patterns among diverse
tabular datasets (tables) and the general scarcity of tabular data available
for fine-tuning. In this study, we fill this gap by introducing a cross-table
pretrained Transformer, XTFormer, for versatile downstream tabular prediction
tasks. Our methodology insight is pretraining XTFormer to establish a
"meta-function" space that encompasses all potential feature-target mappings.
In pre-training, a variety of potential mappings are extracted from
pre-training tabular datasets and are embedded into the "meta-function" space,
and suited mappings are extracted from the "meta-function" space for downstream
tasks by a specified coordinate positioning approach. Experiments show that, in
190 downstream tabular prediction tasks, our cross-table pretrained XTFormer
wins both XGBoost and Catboost on 137 (72%) tasks, and surpasses representative
deep learning models FT-Transformer and the tabular pre-training approach XTab
on 144 (76%) and 162 (85%) tasks.

本研究通过引入一种跨表预训练的 Transformer 模型（XTFormer），填补了现有的跨表数据预测任务中可重复模式有限且数据稀缺的空白。实验结果显示，在 190 项下游表格预测任务中，我们的 XTFormer 在 137 项（72%）任务上胜过 XGBoost 和 Catboost，并且在 144 项（76%）和 162 项（85%）任务上超过了具有代表性的深度学习模型 FT-Transformer 和表格预训练方法 XTab。