Natural language instruction following is paramount to enable collaboration
between artificial agents and human beings. Natural language-conditioned
reinforcement learning (RL) agents have shown how natural languages'
properties, such as compositionality, can provide a strong inductive bias to
learn complex policies. Previous architectures like HIGhER combine the benefit
of language-conditioning with Hindsight Experience Replay (HER) to deal with
sparse rewards environments. Yet, like HER, HIGhER relies on an oracle
predicate function to provide a feedback signal highlighting which linguistic
description is valid for which state. This reliance on an oracle limits its
application. Additionally, HIGhER only leverages the linguistic information
contained in successful RL trajectories, thus hurting its final performance and
data-efficiency. Without early successful trajectories, HIGhER is no better
than DQN upon which it is built. In this paper, we propose the Emergent Textual
Hindsight Experience Replay (ETHER) agent, which builds on HIGhER and addresses
both of its limitations by means of (i) a discriminative visual referential
game, commonly studied in the subfield of Emergent Communication (EC), used
here as an unsupervised auxiliary task and (ii) a semantic grounding scheme to
align the emergent language with the natural language of the
instruction-following benchmark. We show that the referential game's agents
make an artificial language emerge that is aligned with the natural-like
language used to describe goals in the BabyAI benchmark and that it is
expressive enough so as to also describe unsuccessful RL trajectories and thus
provide feedback to the RL agent to leverage the linguistic, structured
information contained in all trajectories. Our work shows that EC is a viable
unsupervised auxiliary task for RL and provides missing pieces to make HER more
widely applicable.

我们提出了 Emergent Textual Hindsight Experience Replay (ETHER) 的代理机制，它从 HIGhER 构建而来，通过 (i) 常见于 Emergent Communication (EC) 领域的判别性视觉指示游戏作为非监督辅助任务以及 (ii) 语义基准方案来解决 HIGhER 的两个局限性，并表明 EC 是 RL 的可行非监督辅助任务，为 HER 的广泛应用提供了遗漏的要点。

ETHER: 对后见体验回放进行新兴通信的调整

ETHER: Aligning Emergent Communication for Hindsight Experience Replay

Language creates a compact representation of the world and allows the
description of unlimited situations and objectives through compositionality.
While these characterizations may foster instructing, conditioning or
structuring interactive agent behavior, it remains an open-problem to correctly
relate language understanding and reinforcement learning in even simple
instruction following scenarios. This joint learning problem is alleviated
through expert demonstrations, auxiliary losses, or neural inductive biases. In
this paper, we propose an orthogonal approach called Hindsight Generation for
Experience Replay (HIGhER) that extends the Hindsight Experience Replay (HER)
approach to the language-conditioned policy setting. Whenever the agent does
not fulfill its instruction, HIGhER learns to output a new directive that
matches the agent trajectory, and it relabels the episode with a positive
reward. To do so, HIGhER learns to map a state into an instruction by using
past successful trajectories, which removes the need to have external expert
interventions to relabel episodes as in vanilla HER. We show the efficiency of
our approach in the BabyAI environment, and demonstrate how it complements
other instruction following methods.

本文提出了一种名为 Hindsight Generation for Experience Replay (HIGhER) 的新方法，其中当智能体不能完成指令时，它学习输出一条新指令来匹配智能体的轨迹，并赋予该回合一个积极的回报，从而消除了需要外部专家干预的需求。在 BabyAI 环境中，我们展示了我们的方法的效率，并展示了它如何补充其他指令跟随方法。