Humans often express their communicative intents indirectly or non-literally,
which requires their interlocutors -- human or AI -- to understand beyond the
literal meaning of words. While most existing work has focused on
discriminative evaluations, we present a new approach to generatively evaluate
large language models' (LLMs') intention understanding by examining their
responses to non-literal utterances. Ideally, an LLM should respond in line
with the true intention of a non-literal utterance, not its literal
interpretation. Our findings show that LLMs struggle to generate pragmatically
relevant responses to non-literal language, achieving only 50-55% accuracy on
average. While explicitly providing oracle intentions significantly improves
performance (e.g., 75% for Mistral-Instruct), this still indicates challenges
in leveraging given intentions to produce appropriate responses. Using
chain-of-thought to make models spell out intentions yields much smaller gains
(60% for Mistral-Instruct). These findings suggest that LLMs are not yet
effective pragmatic interlocutors, highlighting the need for better approaches
for modeling intentions and utilizing them for pragmatic generation.

大语言模型对人类非直接或非文字意义表达的交际意图的理解仍存在挑战，需要更好的方法来建模和利用意图进行实用生成。

人教皇天主教？是的，教皇是天主教的。LLM 中意图解析的生成评估

Is the Pope Catholic? Yes, the Pope is Catholic. Generative Evaluation  of Intent Resolution in LLMs

In order to interpret the communicative intents of an utterance, it needs to
be grounded in something that is outside of language; that is, grounded in
world modalities. In this paper, we argue that dialogue clarification
mechanisms make explicit the process of interpreting the communicative intents
of the speaker's utterances by grounding them in the various modalities in
which the dialogue is situated. This paper frames dialogue clarification
mechanisms as an understudied research problem and a key missing piece in the
giant jigsaw puzzle of natural language understanding. We discuss both the
theoretical background and practical challenges posed by this problem and
propose a recipe for obtaining grounding annotations. We conclude by
highlighting ethical issues that need to be addressed in future work.

本文提出对话澄清机制是自然语言理解中一个未被充分研究的关键问题，并讨论了该问题的理论背景、实际挑战和获取基础注释的方法。同时，还提出了需要在未来研究中解决的伦理问题。