Instruction tuning of open-source large language models (LLMs) like LLaMA,
using direct outputs from more powerful LLMs such as Instruct-GPT and GPT-4,
has proven to be a cost-effective way to align model behaviors with human
preferences. However, the instruction-tuned model has only seen one response
per instruction, lacking the knowledge of potentially better responses. In this
paper, we propose finetuning an instruction-tuned LLM using our novel
\textit{probabilistic ranking} and \textit{contextual ranking} approaches to
increase the likelihood of generating better responses. Probabilistic ranking
enables the instruction-tuned model to inherit the relative rankings of
high-quality and low-quality responses from the teacher LLM. On the other hand,
learning with contextual ranking allows the model to refine its own response
distribution using the contextual understanding ability of stronger LLMs.
Furthermore, we apply probabilistic ranking and contextual ranking sequentially
to the instruction-tuned LLM. The resulting model, which we call \textbf{Tuna},
consistently improves the performance on Super Natural Instructions (119 test
tasks), LMentry (25 test tasks), Vicuna QA, and can even obtain better results
than several strong reinforcement learning baselines. Our code and data are
available at \url{ this https URL}.

通过概率排名和上下文排名的方法对已调整过的模型进行微调，最终生成更好回复的模型称为 Tuna，能够优于强化学习基线模型，提高对各种任务的性能。