Generating diverse and sophisticated instructions for downstream tasks by
Large Language Models (LLMs) is pivotal for advancing the effect. Current
approaches leverage closed-source LLMs, employing in-context prompting for
instruction generation. However, in this paper, we found that in-context
prompting cannot generate complex instructions with length $\ge 100$ for tasks
like code completion.
To solve this problem, we introduce Ada-Instruct, an adaptive instruction
generator developed by fine-tuning open-source LLMs. Our pivotal finding
illustrates that fine-tuning open-source LLMs with a mere ten samples generates
long instructions that maintain distributional consistency for complex
reasoning tasks. We empirically validated Ada-Instruct's efficacy across
different applications, including code completion, mathematical reasoning, and
commonsense reasoning. The results underscore Ada-Instruct's superiority,
evidencing its improvements over its base models, current self-instruct
methods, and other state-of-the-art models.

通过对开源 LLMs 进行微调，我们引入了 Ada-Instruct，一种自适应指令生成器。我们的结果表明，通过仅使用十个样本对开源 LLMs 进行微调，可以生成分布一致的长指令，用于复杂的推理任务。我们在不同的应用中进行了实证验证，包括代码补全、数学推理和常识推理，结果显示 Ada-Instruct 相对于基本模型、当前自我指导方法和其他最先进模型具有优势。