Understanding the non-literal meaning of an utterance is critical for large
language models (LLMs) to become human-like social communicators. In this work,
we introduce SwordsmanImp, the first Chinese multi-turn-dialogue-based dataset
aimed at conversational implicature, sourced from dialogues in the Chinese
sitcom $\textit{My Own Swordsman}$. It includes 200 carefully handcrafted
questions, all annotated on which Gricean maxims have been violated. We test
eight close-source and open-source LLMs under two tasks: a multiple-choice
question task and an implicature explanation task. Our results show that GPT-4
attains human-level accuracy (94%) on multiple-choice questions. CausalLM
demonstrates a 78.5% accuracy following GPT-4. Other models, including GPT-3.5
and several open-source models, demonstrate a lower accuracy ranging from 20%
to 60% on multiple-choice questions. Human raters were asked to rate the
explanation of the implicatures generated by LLMs on their reasonability, logic
and fluency. While all models generate largely fluent and self-consistent text,
their explanations score low on reasonability except for GPT-4, suggesting that
most LLMs cannot produce satisfactory explanations of the implicatures in the
conversation. Moreover, we find LLMs' performance does not vary significantly
by Gricean maxims, suggesting that LLMs do not seem to process implicatures
derived from different maxims differently. Our data and code are available at
this https URL

大规模语言模型的非字面含义对于其成为类似人类的社交交流者至关重要。本文首次介绍了源自中国情景喜剧《武林外传》对话的 Chinese multi-turn-dialogue-based 数据集 SwordsmanImp。我们测试了八种封闭源和开源 LLM 模型在两个任务中的表现：多项选择问题任务和蕴涵说明任务。我们的结果表明 GPT-4 在多项选择问题上达到了人类水平的准确性（94%）。本研究还发现，除了 GPT-4 之外，大多数 LLM 无法对对话中的含义产生令人满意的解释。

大型语言模型是否理解对话言外之意 —— 以中国情景喜剧为例研究

Do Large Language Models Understand Conversational Implicature -- A case  study with a chinese sitcom

The Rational Speech Acts (RSA) model treats language use as a recursive
process in which probabilistic speaker and listener agents reason about each
other's intentions to enrich the literal semantics of their language along
broadly Gricean lines. RSA has been shown to capture many kinds of
conversational implicature, but it has been criticized as an unrealistic model
of speakers, and it has so far required the manual specification of a semantic
lexicon, preventing its use in natural language processing applications that
learn lexical knowledge from data. We address these concerns by showing how to
define and optimize a trained statistical classifier that uses the intermediate
agents of RSA as hidden layers of representation forming a non-linear
activation function. This treatment opens up new application domains and new
possibilities for learning effectively from data. We validate the model on a
referential expression generation task, showing that the best performance is
achieved by incorporating features approximating well-established insights
about natural language generation into RSA.

通过定义和优化经训练的统计分类器，RSA 模型可用作自然语言处理应用程序的隐藏层，从而打开了新的应用领域和有效从数据学习的可能性，我们在指代表达生成任务中验证了该模型，表明将有关自然语言生成的洞见特征纳入 RSA 中可实现最佳性能。