We cast real-world humanoid control as a next token prediction problem, akin
to predicting the next word in language. Our model is a causal transformer
trained via autoregressive prediction of sensorimotor trajectories. To account
for the multi-modal nature of the data, we perform prediction in a
modality-aligned way, and for each input token predict the next token from the
same modality. This general formulation enables us to leverage data with
missing modalities, like video trajectories without actions. We train our model
on a collection of simulated trajectories coming from prior neural network
policies, model-based controllers, motion capture data, and YouTube videos of
humans. We show that our model enables a full-sized humanoid to walk in San
Francisco zero-shot. Our model can transfer to the real world even when trained
on only 27 hours of walking data, and can generalize to commands not seen
during training like walking backward. These findings suggest a promising path
toward learning challenging real-world control tasks by generative modeling of
sensorimotor trajectories.

我们将真实世界中的人形控制视为下一个令牌预测问题，类似于预测语言中的下一个单词。我们的模型是一个因果变换器，通过自回归预测感觉运动轨迹来进行训练。为了考虑数据的多模态性，我们以一种模态对齐的方式进行预测，并且对于每个输入令牌，从相同的模态中预测下一个令牌。这个一般性的公式使我们能够利用缺少模态的数据，例如没有行动的视频轨迹。我们使用来自先前的神经网络策略、基于模型的控制器、动作捕捉数据和人类的 YouTube 视频的模拟轨迹进行模型训练。我们展示了我们的模型使得一个真实尺寸的人形能够在旧金山进行零样本的行走。即使只训练了 27 小时的行走数据，我们的模型也能够在真实世界中实现转移，并且可以推广到在训练期间未见过的命令，如后退行走。这些发现表明通过感觉运动轨迹的生成建模来学习具有挑战性的真实世界控制任务的有前途的路径。