Many real-world applications of language models (LMs), such as code
autocomplete and writing assistance, involve human-LM interaction. However, the
main LM benchmarks are non-interactive in that a system produces output without
human involvement. To evaluate human-LM interaction, we develop a new
framework, Human-AI Language-based Interaction Evaluation (HALIE), that expands
non-interactive evaluation along three dimensions, capturing (i) the
interactive process, not only the final output; (ii) the first-person
subjective experience, not just a third-party assessment; and (iii) notions of
preference beyond quality. We then design five tasks ranging from goal-oriented
to open-ended to capture different forms of interaction. On four
state-of-the-art LMs (three variants of OpenAI's GPT-3 and AI21's J1-Jumbo), we
find that non-interactive performance does not always result in better human-LM
interaction and that first-person and third-party metrics can diverge,
suggesting the importance of examining the nuances of human-LM interaction.

本文通过开发人工智能与自然语言交互评估（HALIE）框架，进行基于人的语言模型交互评估，包括交互过程、第一人称主观体验和执行效果等方面，设计并测试了五种交互式任务，研究了四种先进的语言模型，发现非交互式的性能并不总是导致更好的人工智能与自然语言交互，并且第一人称和第三方评价的指标可以有所区别，因此研究人工智能与自然语言交互的微妙差别非常重要。