While large language models (LLMs) bring not only performance but also
complexity, recent work has started to turn LLMs into data generators rather
than task inferencers, where another affordable task model is trained for
efficient deployment and inference. However, such an approach has primarily
been applied to natural language tasks and has not yet been explored for
symbolic language tasks with complex structured outputs (e.g., semantic parsing
and code generation). In this paper, we propose SymGen which utilizes LLMs for
generating various annotation-expensive symbolic language data. SymGen consists
of an informative prompt to steer generation and an agreement-based verifier to
improve data correctness. We conduct extensive experiments on six symbolic
language tasks across various settings. Compared with the LLMs, we demonstrate
the 1\%-sized task model can achieve comparable or better performance, largely
cutting inference and deployment costs. We also show that generated data with
only a few human demonstrations can be as effective as over 10 times the amount
of human-annotated data when training the task model, saving a considerable
amount of annotation effort. SymGen sheds new light on data generation for
complex tasks, and we release the code at
\href{https://github.com/HKUNLP/SymGen}{this https URL}.

本篇论文提出 SymGen，一种使用 LLMs 生成注释昂贵的符号语言数据的方法。与 LLMs 相比，SymGen 可以生成与人类注释数据效果相当或更好的数据，同时节省了大量的注释工作量。