To interact with humans in the world, agents need to understand the diverse
types of language that people use, relate them to the visual world, and act
based on them. While current agents learn to execute simple language
instructions from task rewards, we aim to build agents that leverage diverse
language that conveys general knowledge, describes the state of the world,
provides interactive feedback, and more. Our key idea is that language helps
agents predict the future: what will be observed, how the world will behave,
and which situations will be rewarded. This perspective unifies language
understanding with future prediction as a powerful self-supervised learning
objective. We present Dynalang, an agent that learns a multimodal world model
that predicts future text and image representations and learns to act from
imagined model rollouts. Unlike traditional agents that use language only to
predict actions, Dynalang acquires rich language understanding by using past
language also to predict future language, video, and rewards. In addition to
learning from online interaction in an environment, Dynalang can be pretrained
on datasets of text, video, or both without actions or rewards. From using
language hints in grid worlds to navigating photorealistic scans of homes,
Dynalang utilizes diverse types of language to improve task performance,
including environment descriptions, game rules, and instructions.

为了与人类进行互动，代理机构需要理解人们使用的各种语言类型，将其与视觉世界联系起来，并根据这些语言进行行动。本文提出了 Dynalang，这是一种代理机构，它学习了一个多模态世界模型，预测未来的文本和图像表示，并学会根据想象的模型展开行动。与传统代理机构只使用语言预测行动不同，Dynalang 通过使用过去的语言预测未来的语言、视频和奖励，获得了丰富的语言理解能力。