Tabular data is a pervasive modality spanning a wide range of domains, and
the inherent diversity poses a considerable challenge for deep learning. Recent
advancements using transformer-based in-context learning have shown promise on
smaller and less complex datasets, but have struggled to scale to larger and
more complex ones. To address this limitation, we propose a combination of
retrieval and fine-tuning: we can adapt the transformer to a local subset of
the data by collecting nearest neighbours, and then perform task-specific
fine-tuning with this retrieved set of neighbours in context. Using TabPFN as
the base model -- currently the best tabular in-context learner -- and applying
our retrieval and fine-tuning scheme on top results in what we call a
locally-calibrated PFN, or LoCalPFN. We conduct extensive evaluation on 95
datasets curated by TabZilla from OpenML, upon which we establish a new
state-of-the-art with LoCalPFN -- even with respect to tuned tree-based models.
Notably, we show a significant boost in performance compared to the base
in-context model, demonstrating the efficacy of our approach and advancing the
frontier of deep learning in tabular data.

基于表格数据的局部校准 PFN 模型，在 TabZilla 的 95 个数据集上，提供了改进的性能，从而展示了我们方法对于深度学习在表格数据中的前沿研究所取得的有效性。