Instruction tuning -- fine-tuning a large language model (LLM) on pairs of
instructions and desired outcomes -- is an approach that enables pre-trained
language models to perform real-world tasks and follow human instructions. Its
practical success depends on the model learning a broader set of instructions
than those it was trained on. Yet the factors that determine model
generalization to such \emph{unseen tasks} are not well understood. %To
understand the driving factors of generalization, In this paper, we experiment
with string rewrites, a symbolic task that serves as a building block for
Turing complete Markov algorithms while allowing experimental control of
"inputs" and "instructions". We investigate the trade-off between the number of
instructions the model is trained on and the number of training samples
provided for each instruction and observe that the diversity of the instruction
set determines generalization. Generalization emerges once a diverse enough set
of tasks is provided, even though very few examples are provided for each task.
Instruction diversity also ensures robustness with respect to non-uniform
distributions of instructions in the training set.

通过调整指令，大型语言模型（LLM）可以进行微调，实现真实世界任务执行和遵循人类指令；实现普遍使用的关键取决于模型学习更广泛的指令集而不仅仅是训练集上的指令。本文通过实验探究了指令多样性与模型普适性之间的权衡，发现多样性的指令集使模型具备普适性，并确保在训练集上非均匀分布指令的情况下保持鲁棒性。