Large language models (LLMs) adapted to follow user instructions are now
widely deployed as conversational agents. In this work, we examine one
increasingly common instruction-following task: providing writing assistance to
compose a long-form answer. To evaluate the capabilities of current LLMs on
this task, we construct KIWI, a dataset of knowledge-intensive writing
instructions in the scientific domain. Given a research question, an initial
model-generated answer and a set of relevant papers, an expert annotator
iteratively issues instructions for the model to revise and improve its answer.
We collect 1,260 interaction turns from 234 interaction sessions with three
state-of-the-art LLMs. Each turn includes a user instruction, a model response,
and a human evaluation of the model response. Through a detailed analysis of
the collected responses, we find that all models struggle to incorporate new
information into an existing answer, and to perform precise and unambiguous
edits. Further, we find that models struggle to judge whether their outputs
successfully followed user instructions, with accuracy at least 10 points short
of human agreement. Our findings indicate that KIWI will be a valuable resource
to measure progress and improve LLMs' instruction-following capabilities for
knowledge intensive writing tasks.

通过对 LLMs 在写作辅助任务上的能力进行评估，我们发现所有模型在将新信息纳入现有答案以及进行精确和明确编辑方面都有困难。此外，我们发现模型在判断其输出是否成功遵循用户指令方面也存在困难。研究结果表明，KIWI 将成为衡量进展和改进 LLMs 在知识密集型写作任务中的指令跟随能力的宝贵资源。