We present a method for combining multi-agent communication and traditional
data-driven approaches to natural language learning, with an end goal of
teaching agents to communicate with humans in natural language. Our starting
point is a language model that has been trained on generic, not task-specific
language data. We then place this model in a multi-agent self-play environment
that generates task-specific rewards used to adapt or modulate the model,
turning it into a task-conditional language model. We introduce a new way for
combining the two types of learning based on the idea of reranking language
model samples, and show that this method outperforms others in communicating
with humans in a visual referential communication task. Finally, we present a
taxonomy of different types of language drift that can occur alongside a set of
measures to detect them.

本文介绍一种结合多智能体通信和传统数据驱动方法的自然语言学习方法，通过在自身玩耍的环境中生成任务特定的奖励来适应或调节模型，形成任务条件化语言模型，并引入了一种基于语言模型样本重新排序的新方法，以优于其他方法地与人类进行视觉指称交流任务的通信。最后，我们提出了一种不同类型的语言漂移分类以及检测它们的措施。

多智能体通信与自然语言：功能与结构语言学之间的协同

Multi-agent Communication meets Natural Language: Synergies between  Functional and Structural Language Learning

Pretraining on human corpus and then finetuning in a simulator has become a
standard pipeline for training a goal-oriented dialogue agent. Nevertheless, as
soon as the agents are finetuned to maximize task completion, they suffer from
the so-called language drift phenomenon: they slowly lose syntactic and
semantic properties of language as they only focus on solving the task. In this
paper, we propose a generic approach to counter language drift called Seeded
iterated learning (SIL). We periodically refine a pretrained student agent by
imitating data sampled from a newly generated teacher agent. At each time step,
the teacher is created by copying the student agent, before being finetuned to
maximize task completion. SIL does not require external syntactic constraint
nor semantic knowledge, making it a valuable task-agnostic finetuning protocol.
We evaluate SIL in a toy-setting Lewis Game, and then scale it up to the
translation game with natural language. In both settings, SIL helps counter
language drift as well as it improves the task completion compared to
baselines.

本文提出了一种通用的方法来对抗所谓的语言漂移现象，这种方法叫做 Seeded iterated learning (SIL)，该方法通过周期性地对预训练的学生智能体进行附着，以模仿从新生成的老师智能体采样的数据，并评估了该方法的效果。

种子迭代学习抵制语言漂移

Countering Language Drift with Seeded Iterated Learning

Emergent multi-agent communication protocols are very different from natural
language and not easily interpretable by humans. We find that agents that were
initially pretrained to produce natural language can also experience
detrimental language drift: when a non-linguistic reward is used in a
goal-based task, e.g. some scalar success metric, the communication protocol
may easily and radically diverge from natural language. We recast translation
as a multi-agent communication game and examine auxiliary training constraints
for their effectiveness in mitigating language drift. We show that a
combination of syntactic (language model likelihood) and semantic (visual
grounding) constraints gives the best communication performance, allowing
pre-trained agents to retain English syntax while learning to accurately convey
the intended meaning.

通过在多智能体通信中引入语言模型和视觉约束等训练约束，可以有效避免非语言性的奖励对预先训练的代理产生的语言漂移，从而使预先训练的代理在保留英语语法的同时学会准确传达信息。