Recent developments show that Large Language Models (LLMs) produce state-of-the-art performance on natural language (NL) to code generation for resource-rich general-purpose languages like C++, Java, and Python. However, their practical usage for structured domain-specific languages (DSLs) such as YAML, JSON is limited due to domain-specific schema, grammar, and customizations generally unseen by LLMs during pre-training. Efforts have been made to mitigate this challenge via in-context learning through relevant examples or by fine-tuning. However, it suffers from problems, such as limited DSL samples and prompt sensitivity but enterprises maintain good documentation of the DSLs. Therefore, we propose DocCGen, a framework that can leverage such rich knowledge by breaking the NL-to-Code generation task for structured code languages into a two-step process. First, it detects the correct libraries using the library documentation that best matches the NL query. Then, it utilizes schema rules extracted from the documentation of these libraries to constrain the decoding. We evaluate our framework for two complex structured languages, Ansible YAML and Bash command, consisting of two settings: Out-of-domain (OOD) and In-domain (ID). Our extensive experiments show that DocCGen consistently improves different-sized language models across all six evaluation metrics, reducing syntactic and semantic errors in structured code. We plan to open-source the datasets and code to motivate research in constrained code generation.

大型语言模型在自然语言到代码生成方面展现出卓越的性能，在通用领域语言（如C++、Java和Python）上表现出最先进的性能。但是，在面向特定领域的结构化语言（如YAML、JSON）上，它们的实际使用受到限制。为了克服这个挑战，我们提出了DocCGen框架，通过将结构化代码语言的自然语言到代码生成任务分为两步处理。通过从库文档中提取的模式规则对解码进行约束，我们的框架在两个复杂的结构化语言（Ansible YAML和Bash命令）上不断改进不同规模的语言模型，并降低了结构化代码中的语法和语义错误。我们计划开源数据集和代码以促进受限制代码生成的研究。

DocCGen: 基于文档的受控代码生成