This paper evaluates the extent to which current Large Language Models (LLMs)
can capture task-oriented multi-party conversations (MPCs). We have recorded
and transcribed 29 MPCs between patients, their companions, and a social robot
in a hospital. We then annotated this corpus for multi-party goal-tracking and
intent-slot recognition. People share goals, answer each other's goals, and
provide other people's goals in MPCs - none of which occur in dyadic
interactions. To understand user goals in MPCs, we compared three methods in
zero-shot and few-shot settings: we fine-tuned T5, created pre-training tasks
to train DialogLM using LED, and employed prompt engineering techniques with
GPT-3.5-turbo, to determine which approach can complete this novel task with
limited data. GPT-3.5-turbo significantly outperformed the others in a few-shot
setting. The `reasoning' style prompt, when given 7% of the corpus as example
annotated conversations, was the best performing method. It correctly annotated
62.32% of the goal tracking MPCs, and 69.57% of the intent-slot recognition
MPCs. A `story' style prompt increased model hallucination, which could be
detrimental if deployed in safety-critical settings. We conclude that
multi-party conversations still challenge state-of-the-art LLMs.

通过对大规模语言模型在任务导向的多方对话中捕捉能力的评估，本研究记录并转录了医院中患者、他们的陪伴者和社交机器人之间的 29 个多方对话，并对此语料库进行了多方目标跟踪和意图 - 插槽识别的注释。我们在零样本和少样本设置中比较了三种方法，即微调 T5，使用 LED 创建预训练任务以训练 DialogLM，并采用 GPT-3.5-turbo 的提示工程技术，以确定哪种方法可以用有限数据完成这一新任务。在少样本设置中，GPT-3.5-turbo 显著优于其他方法。当给出例子注释对话中 7% 的语料库时，“推理” 风格的提示是效果最好的方法。它正确注释了 62.32% 的目标跟踪多方对话和 69.57% 的意图 - 插槽识别多方对话。而 “故事” 风格的提示会增加模型的虚构成分，在安全关键环境中可能会有不利影响。我们得出结论，多方对话仍然对最先进的大规模语言模型构成挑战。