Understanding the non-literal meaning of an utterance is critical for large
language models (LLMs) to become human-like social communicators. In this work,
we introduce SwordsmanImp, the first Chinese multi-turn-dialogue-based dataset
aimed at conversational implicature, sourced from dialogues in the Chinese
sitcom $\textit{My Own Swordsman}$. It includes 200 carefully handcrafted
questions, all annotated on which Gricean maxims have been violated. We test
eight close-source and open-source LLMs under two tasks: a multiple-choice
question task and an implicature explanation task. Our results show that GPT-4
attains human-level accuracy (94%) on multiple-choice questions. CausalLM
demonstrates a 78.5% accuracy following GPT-4. Other models, including GPT-3.5
and several open-source models, demonstrate a lower accuracy ranging from 20%
to 60% on multiple-choice questions. Human raters were asked to rate the
explanation of the implicatures generated by LLMs on their reasonability, logic
and fluency. While all models generate largely fluent and self-consistent text,
their explanations score low on reasonability except for GPT-4, suggesting that
most LLMs cannot produce satisfactory explanations of the implicatures in the
conversation. Moreover, we find LLMs' performance does not vary significantly
by Gricean maxims, suggesting that LLMs do not seem to process implicatures
derived from different maxims differently. Our data and code are available at
this https URL

大规模语言模型的非字面含义对于其成为类似人类的社交交流者至关重要。本文首次介绍了源自中国情景喜剧《武林外传》对话的 Chinese multi-turn-dialogue-based 数据集 SwordsmanImp。我们测试了八种封闭源和开源 LLM 模型在两个任务中的表现：多项选择问题任务和蕴涵说明任务。我们的结果表明 GPT-4 在多项选择问题上达到了人类水平的准确性（94%）。本研究还发现，除了 GPT-4 之外，大多数 LLM 无法对对话中的含义产生令人满意的解释。

大型语言模型是否理解对话言外之意 —— 以中国情景喜剧为例研究

Do Large Language Models Understand Conversational Implicature -- A case  study with a chinese sitcom

NLP tasks are typically defined extensionally through datasets containing
example instantiations (e.g., pairs of image i and text t), but motivated
intensionally through capabilities invoked in verbal descriptions of the task
(e.g., "t is a description of i, for which the content of i needs to be
recognised and understood"). We present Pento-DIARef, a diagnostic dataset in a
visual domain of puzzle pieces where referring expressions are generated by a
well-known symbolic algorithm (the "Incremental Algorithm"), which itself is
motivated by appeal to a hypothesised capability (eliminating distractors
through application of Gricean maxims). Our question then is whether the
extensional description (the dataset) is sufficient for a neural model to pick
up the underlying regularity and exhibit this capability given the simple task
definition of producing expressions from visual inputs. We find that a model
supported by a vision detection step and a targeted data generation scheme
achieves an almost perfect BLEU@1 score and sentence accuracy, whereas simpler
baselines do not.

本文介绍了一种基于符号算法和谈判理论的人工智能推理模型，评估了它对于语言任务的处理能力，结果显示此模型表现良好，并提出了更好的模型设计支撑。