Synthetic tabular data generation with differential privacy is a crucial
problem to enable data sharing with formal privacy. Despite a rich history of
methodological research and development, developing differentially private
tabular data generators that can provide realistic synthetic datasets remains
challenging. This paper introduces DP-LLMTGen -- a novel framework for
differentially private tabular data synthesis that leverages pretrained large
language models (LLMs). DP-LLMTGen models sensitive datasets using a two-stage
fine-tuning procedure with a novel loss function specifically designed for
tabular data. Subsequently, it generates synthetic data through sampling the
fine-tuned LLMs. Our empirical evaluation demonstrates that DP-LLMTGen
outperforms a variety of existing mechanisms across multiple datasets and
privacy settings. Additionally, we conduct an ablation study and several
experimental analyses to deepen our understanding of LLMs in addressing this
important problem. Finally, we highlight the controllable generation ability of
DP-LLMTGen through a fairness-constrained generation setting.

使用预训练的大型语言模型（LLMs），DP-LLMTGen 引入了一种新的框架来进行差分私有表格数据合成，通过微调过程模拟敏感数据集并生成多样的合成数据，有效地解决了实际挑战。