While tabular classification has traditionally relied on from-scratch
training, a recent breakthrough called prior-data fitted networks (PFNs)
challenges this approach. Similar to large language models, PFNs make use of
pretraining and in-context learning to achieve strong performance on new tasks
in a single forward pass. However, current PFNs have limitations that prohibit
their widespread adoption. Notably, TabPFN achieves very strong performance on
small tabular datasets but is not designed to make predictions for datasets of
size larger than 1000. In this work, we overcome these limitations and
substantially improve the performance of PFNs by developing context
optimization techniques for PFNs. Specifically, we propose TuneTables, a novel
prompt-tuning strategy that compresses large datasets into a smaller learned
context. TuneTables scales TabPFN to be competitive with state-of-the-art
tabular classification methods on larger datasets, while having a substantially
lower inference time than TabPFN. Furthermore, we show that TuneTables can be
used as an interpretability tool and can even be used to mitigate biases by
optimizing a fairness objective.

通过开发上下文优化技术，我们提出了一种名为 TuneTables 的新型提示调整策略，将大数据集压缩成较小的学习上下文，从而显著提高了 PFN 的性能，并在较大数据集上与最先进的表格分类方法具有竞争力，同时推断时间大幅降低。此外，我们展示了 TuneTables 作为一种可解释性工具的应用，并通过优化公正目标来减轻偏见问题。