Instruction tuning -- tuning large language models on instruction-output
pairs -- is a promising technique for making models better adapted to the real
world. Yet, the key factors driving the model's capability to understand and
follow instructions not seen during training remain under-explored. Our
investigation begins with a series of synthetic experiments within the
theoretical framework of a Turing-complete algorithm called Markov algorithm,
which allows fine-grained control over the instruction-tuning data.
Generalization and robustness with respect to the training distribution emerge
once a diverse enough set of tasks is provided, even though very few examples
are provided for each task. We extend these initial results to a real-world
application scenario of code generation and find that a more diverse
instruction set, extending beyond code-related tasks, improves the performance
of code generation. Our observations suggest that a more diverse semantic space
for instruction-tuning sets greatly improves the model's ability to follow
instructions and perform tasks.

通过合适的指令调整，语言模型在指令 - 输出对的训练上能够更好地适应真实世界；研究发现，提供足够多样化的任务集合能够使其在训练分布之外的情况下具备泛化和鲁棒性，并且扩展调整指令集合能够提高代码生成的性能。

从符号任务到代码生成：多样化产生更好的任务执行者

From Symbolic Tasks to Code Generation: Diversification Yields Better  Task Performers

Building LLMs for languages other than English is in great demand due to the
unavailability and performance of multilingual LLMs, such as understanding the
local context. The problem is critical for low-resource languages due to the
need for instruction sets. In a multilingual country like India, there is a
need for LLMs supporting Indic languages to provide generative AI and LLM-based
technologies and services to its citizens.
This paper presents our approach of i) generating a large Odia instruction
set, including domain knowledge data suitable for LLM fine-tuning, and ii)
building a Llama2-finetuned model tailored for enhanced performance in the Odia
domain. The proposed work will help researchers build an instruction set and
LLM, particularly for Indic languages. We will release the model and
instruction set for the public for research and noncommercial purposes.

本文介绍了我们生成大量 Odia 指令集以及领域知识数据，并构建针对 Odia 领域性能优化的 Llama2 微调模型的方法，这将帮助研究人员构建指令集和 LLM，特别是针对 Indic 语言。我们将为公众提供该模型和指令集，供研究和非商业用途使用。