Given the ubiquitous use of tabular data in industries and the growing
concerns in data privacy and security, tabular data synthesis emerges as a
critical research area. The recent state-of-the-art methods show that large
language models (LLMs) can be adopted to generate realistic tabular data. As
LLMs pre-process tabular data as full text, they have the advantage of avoiding
the curse of dimensionality associated with one-hot encoding high-dimensional
data. However, their long training time and limited re-usability on new tasks
prevent them from replacing exiting tabular generative models. In this paper,
we propose Tabula, a tabular data synthesizer based on the language model
structure. Through Tabula, we demonstrate the inherent limitation of employing
pre-trained language models designed for natural language processing (NLP) in
the context of tabular data synthesis. Our investigation delves into the
development of a dedicated foundational model tailored specifically for tabular
data synthesis. Additionally, we propose a token sequence compression strategy
to significantly reduce training time while preserving the quality of synthetic
data. Extensive experiments on six datasets demonstrate that using a language
model structure without loading the well-trained model weights yields a better
starting model for tabular data synthesis. Moreover, the Tabula model,
previously trained on other tabular data, serves as an excellent foundation
model for new tabular data synthesis tasks. Additionally, the token sequence
compression method substantially reduces the model's training time. Results
show that Tabula averagely reduces 46.2% training time per epoch comparing to
current LLMs-based state-of-the-art algorithm and consistently achieves even
higher synthetic data utility.

通过提出 Tabula，使用语言模型结构的表格数据合成器，我们揭示了使用为自然语言处理设计的预训练语言模型在表格数据合成领域的内在局限性，并通过一种令人满意的基础模型开展了针对表格数据合成的专用基础模型的研发。此外，我们提出了一种令牌序列压缩策略，可显著减少训练时间同时保持合成数据的质量。实验证明，使用语言模型结构而不加载训练有素的模型权重可以获得更好的表格数据合成起始模型。此外，之前针对其他表格数据训练的 Tabula 模型可作为新的表格数据合成任务的优秀基础模型，而且该令牌序列压缩方法可大幅降低模型的训练时间。结果表明，Tabula 相较于当前基于 LLMs 的最先进算法而言，平均每个时期减少 46.2% 的训练时间，并始终获得更高的合成数据效用。