Previous in-context learning (ICL) research has focused on tasks such as
classification, machine translation, text2table, etc., while studies on whether
ICL can improve human-like dialogue generation are scarce. Our work fills this
gap by systematically investigating the ICL capabilities of large language
models (LLMs) in persona-based dialogue generation, conducting extensive
experiments on high-quality real human Chinese dialogue datasets. From
experimental results, we draw three conclusions: 1) adjusting prompt
instructions is the most direct, effective, and economical way to improve
generation quality; 2) randomly retrieving demonstrations (demos) achieves the
best results, possibly due to the greater diversity and the amount of effective
information; counter-intuitively, retrieving demos with a context identical to
the query performs the worst; 3) even when we destroy the multi-turn
associations and single-turn semantics in the demos, increasing the number of
demos still improves dialogue performance, proving that LLMs can learn from
corrupted dialogue demos. Previous explanations of the ICL mechanism, such as
$n$-gram induction head, cannot fully account for this phenomenon.

通过在高质量的真实人类中文对话数据集上对大型语言模型（LLMs）在基于角色的对话生成中的上下文学习能力进行广泛实验，我们的工作填补了上下文学习研究在人类对话生成领域的空白。从实验结果中，我们得出了三个结论：1）调整提示指令是改善生成质量最直接、最有效、最经济的方法；2）随机检索演示（demos）可以取得最好的结果，可能是因为其更大的多样性和有效信息的数量；与查询上下文完全相同的检索演示的表现最差，与直觉相反；3）即使在演示中破坏了多轮关联和单轮语义，增加演示数量仍然可以提高对话性能，证明了 LLMs 可以从损坏的对话演示中进行学习。先前对 ICL 机制的解释，例如 $n$-gram 归纳头，不能完全解释这一现象。