Foundation models have revolutionized tasks in computer vision and natural
language processing. However, in the realm of tabular data, tree-based models
like XGBoost continue to dominate. TabPFN, a transformer model tailored for
tabular data, mirrors recent foundation models in its exceptional in-context
learning capability, being competitive with XGBoost's performance without the
need for task-specific training or hyperparameter tuning. Despite its promise,
TabPFN's applicability is hindered by its data size constraint, limiting its
use in real-world scenarios. To address this, we present in-context data
distillation (ICD), a novel methodology that effectively eliminates these
constraints by optimizing TabPFN's context. ICD efficiently enables TabPFN to
handle significantly larger datasets with a fixed memory budget, improving
TabPFN's quadratic memory complexity but at the cost of a linear number of
tuning steps. Notably, TabPFN, enhanced with ICD, demonstrates very strong
performance against established tree-based models and modern deep learning
methods on 48 large tabular datasets from OpenML.

TabPFN 通过采用 in-context data distillation 方法，优化了原有模型的上下文效果，使其能够处理大规模数据集，从而在 48 个大型表格数据集上展示了比基于树模型和现代深度学习方法更强的性能。