Deep reinforcement learning agents need to be trained over millions of
episodes to decently solve navigation tasks grounded to instructions.
Furthermore, their ability to generalize to novel combinations of instructions
is unclear. Interestingly however, children can decompose language-based
instructions and navigate to the referred object, even if they have not seen
the combination of queries prior. Hence, we created three 3D environments to
investigate how deep RL agents learn and compose color-shape based
combinatorial instructions to solve novel combinations in a spatial navigation
task. First, we explore if agents can perform compositional learning, and
whether they can leverage on frozen text encoders (e.g. CLIP, BERT) to learn
word combinations in fewer episodes. Next, we demonstrate that when agents are
pretrained on the shape or color concepts separately, they show a 20 times
decrease in training episodes needed to solve unseen combinations of
instructions. Lastly, we show that agents pretrained on concept and
compositional learning achieve significantly higher reward when evaluated
zero-shot on novel color-shape1-shape2 visual object combinations. Overall, our
results highlight the foundations needed to increase an agent's proficiency in
composing word groups through reinforcement learning and its ability for
zero-shot generalization to new combinations.

深度强化学习代理需要数百万次训练以解决与指令相关的导航任务，对于其泛化到新颖指令组合的能力还不清楚。然而，有趣的是，儿童能够将基于语言的指令分解并导航到所指对象，即使他们以前没有见过这种查询的组合。因此，我们创建了三个 3D 环境来研究深度强化学习代理如何学习和组合基于颜色和形状的组合指令来解决空间导航任务中的新颖组合。首先，我们探索代理是否能进行组合学习，并且它们是否可以利用冻结的文本编码器（例如 CLIP，BERT）在较少的训练次数内学习词组合。接下来，我们证明当代理在形状或颜色概念上预训练时，它们解决未见组合指令所需的训练次数减少了 20 倍。最后，我们展示了在概念和组合学习上进行预训练的代理在评估时的零样本状态下，对新的颜色 - 形状 1 - 形状 2 视觉对象组合获得了显著更高的奖励。总体而言，我们的结果凸显了通过强化学习增强代理组合词语的能力以及其对新组合的零样本泛化能力所需的基础。