Foundational models (FMs), pretrained on extensive datasets using
self-supervised techniques, are capable of learning generalized patterns from
large amounts of data. This reduces the need for extensive labeled datasets for
each new task, saving both time and resources by leveraging the broad knowledge
base established during pretraining. Most research on FMs has primarily focused
on unstructured data, such as text and images, or semi-structured data, like
time-series. However, there has been limited attention to structured data, such
as tabular data, which, despite its prevalence, remains under-studied due to a
lack of clean datasets and insufficient research on the transferability of FMs
for various tabular data tasks. In response to this gap, we introduce a
framework called TabularFM (https://tabularfm.github.io/), which
incorporates state-of-the-art methods for developing FMs specifically for
tabular data. This includes variations of neural architectures such as GANs,
VAEs, and Transformers. We have curated a million of tabular datasets and
released cleaned versions to facilitate the development of tabular FMs. We
pretrained FMs on this curated data, benchmarked various learning methods on
these datasets, and released the pretrained models along with leaderboards for
future comparative studies. Our fully open-sourced system provides a
comprehensive analysis of the transferability of tabular FMs. By releasing
these datasets, pretrained models, and leaderboards, we aim to enhance the
validity and usability of tabular FMs in the near future.

TabularFM 是一个针对结构化数据的框架，通过预训练的方式使用 GANs、VAEs 和 Transformers 等神经结构提高表格数据模型的可迁移性，并提供百万个经过清洗的表格数据集、预训练模型和排行榜，以增强表格数据模型的有效性和可用性。