Learning on tabular data underpins numerous real-world applications. Despite considerable efforts in developing effective learning models for tabular data, current transferable tabular models remain in their infancy, limited by either the lack of support for direct instruction following in new tasks or the neglect of acquiring foundational knowledge and capabilities from diverse tabular datasets. In this paper, we propose Tabular Foundation Models (TabFMs) to overcome these limitations. TabFMs harness the potential of generative tabular learning, employing a pre-trained large language model (LLM) as the base model and fine-tuning it using purpose-designed objectives on an extensive range of tabular datasets. This approach endows TabFMs with a profound understanding and universal capabilities essential for learning on tabular data. Our evaluations underscore TabFM's effectiveness: not only does it significantly excel in instruction-following tasks like zero-shot and in-context inference, but it also showcases performance that approaches, and in instances, even transcends, the renowned yet mysterious closed-source LLMs like GPT-4. Furthermore, when fine-tuning with scarce data, our model achieves remarkable efficiency and maintains competitive performance with abundant training data. Finally, while our results are promising, we also delve into TabFM's limitations and potential opportunities, aiming to stimulate and expedite future research on developing more potent TabFMs.

本文提出了Tabular Foundation Models (TabFMs)，通过在广泛的表格数据集上使用预训练的大型语言模型 (LLM) 并进行微调，实现了对表格数据具有深刻理解和普适能力的目标。TabFMs在指导性任务（如零样本和上下文推理）方面具有显著优势，并且在某些情况下甚至超越了著名但神秘的闭源LLMs，如GPT-4。此外，当仅有有限的数据进行微调时，我们的模型表现出了出色的效率和竞争性表现。最后，我们也探讨了TabFM的局限性和潜在机会，旨在激发和促进未来开发更强大的TabFMs的研究。

面向表格数据学习的基础模型研究