Dialogue participants may have varying levels of knowledge about the topic
under discussion. In such cases, it is essential for speakers to adapt their
utterances by taking their audience into account. Yet, it is an open question
how such adaptation can be modelled in computational agents. In this paper, we
model a visually grounded referential game between a knowledgeable speaker and
a listener with more limited visual and linguistic experience. Inspired by
psycholinguistic theories, we endow our speaker with the ability to adapt its
referring expressions via a simulation module that monitors the effectiveness
of planned utterances from the listener's perspective. We propose an adaptation
mechanism building on plug-and-play approaches to controlled language
generation, where utterance generation is steered on the fly by the simulator
without finetuning the speaker's underlying language model. Our results and
analyses show that our approach is effective: the speaker's utterances become
closer to the listener's domain of expertise, which leads to higher
communicative success.

本文提出一种基于心理语言学理论的适应性模型，通过模拟模块监控计划发言的有效性并实时控制话语生成，实现知识丰富的说话者对具有更有限语言和视觉经验的听者进行言语适应，以提高沟通成功率。

听众意识调适：透过即插即用的心理理论，说出符合听众语言的话

Speaking the Language of Your Listener: Audience-Aware Adaptation via  Plug-and-Play Theory of Mind

Human language users can generate descriptions of perceptual concepts beyond
instance-level representations and also use such descriptions to learn
provisional class-level representations. However, the ability of computational
models to learn and operate with class representations is under-investigated in
the language-and-vision field. In this paper, we train separate neural networks
to generate and interpret class-level descriptions. We then use the zero-shot
classification performance of the interpretation model as a measure of
communicative success and class-level conceptual grounding. We investigate the
performance of prototype- and exemplar-based neural representations grounded
category description. Finally, we show that communicative success reveals
performance issues in the generation model that are not captured by traditional
intrinsic NLG evaluation metrics, and argue that these issues can be traced to
a failure to properly ground language in vision at the class level. We observe
that the interpretation model performs better with descriptions that are low in
diversity on the class level, possibly indicating a strong reliance on
frequently occurring features.

本文探究了计算模型学习和运用类别表征的能力，使用零样本分类推断来度量交际成功和类别概念基础，并通过两种不同的神经表征（基于原型和范例）研究了类别描述的植入性。结果发现属性重复度高的描述更容易达到交际成功，而这也暴露了类别描述生成模型中存在的性能问题。

描述奥凯特：生成基于感知的类别描述

Describe me an Aucklet: Generating Grounded Perceptual Category Descriptions

An overarching goal of natural language processing is to enable machines to
communicate seamlessly with humans. However, natural language can be ambiguous
or unclear. In cases of uncertainty, humans engage in an interactive process
known as repair: asking questions and seeking clarification until their
uncertainty is resolved. We propose a framework for building a visually
grounded question-asking model capable of producing polar (yes-no)
clarification questions to resolve misunderstandings in dialogue. Our model
uses an expected information gain objective to derive informative questions
from an off-the-shelf image captioner without requiring any supervised
question-answer data. We demonstrate our model's ability to pose questions that
improve communicative success in a goal-oriented 20 questions game with
synthetic and human answerers.

通过提出一个视觉基础的问题 - 回答模型框架，使用期望的信息量增益优化来产生极性澄清问题以消除人机对话中的误解，从而展示了该模型在目标导向的 20 个问题游戏中与人工回答者一起提出的问题如何提高交流成功率。