Imitation learning learns a policy from demonstrations without requiring
hand-designed reward functions. In many robotic tasks, such as autonomous
racing, imitated policies must model complex environment dynamics and human
decision-making. Sequence modeling is highly effective in capturing intricate
patterns of motion sequences but struggles to adapt to new environments or
distribution shifts that are common in real-world robotics tasks. In contrast,
Adversarial Imitation Learning (AIL) can mitigate this effect, but struggles
with sample inefficiency and handling complex motion patterns. Thus, we propose
BeTAIL: Behavior Transformer Adversarial Imitation Learning, which combines a
Behavior Transformer (BeT) policy from human demonstrations with online AIL.
BeTAIL adds an AIL residual policy to the BeT policy to model the sequential
decision-making process of human experts and correct for out-of-distribution
states or shifts in environment dynamics. We test BeTAIL on three challenges
with expert-level demonstrations of real human gameplay in Gran Turismo Sport.
Our proposed residual BeTAIL reduces environment interactions and improves
racing performance and stability, even when the BeT is pretrained on different
tracks than downstream learning. Videos and code available at:
this https URL

BeTAIL 是一种结合行为转换（BeT）和在线对抗仿真学习（AIL）的方法，用于建模人类专家的顺序决策流程并纠正动态环境状态的变化，从而改善赛车性能和稳定性。

BeTAIL: 人工赛车游戏中基于行为转换的对抗模仿学习

BeTAIL: Behavior Transformer Adversarial Imitation Learning from Human  Racing Gameplay

While large-scale sequence modeling from offline data has led to impressive
performance gains in natural language and image generation, directly
translating such ideas to robotics has been challenging. One critical reason
for this is that uncurated robot demonstration data, i.e. play data, collected
from non-expert human demonstrators are often noisy, diverse, and
distributionally multi-modal. This makes extracting useful, task-centric
behaviors from such data a difficult generative modeling problem. In this work,
we present Conditional Behavior Transformers (C-BeT), a method that combines
the multi-modal generation ability of Behavior Transformer with
future-conditioned goal specification. On a suite of simulated benchmark tasks,
we find that C-BeT improves upon prior state-of-the-art work in learning from
play data by an average of 45.7%. Further, we demonstrate for the first time
that useful task-centric behaviors can be learned on a real-world robot purely
from play data without any task labels or reward information. Robot videos are
best viewed on our project website: this https URL

本文提出了一种名为 C-BeT 的新方法，结合了 Behavior Transformer 多模态生成能力和条件生成模型技术，实现从多模态、嘈杂数据中提取有用、任务中心的行为，取得了有益的效果，而该方法也是第一个实现从无标签、零奖励信息的玩耍数据里纯学习真实世界机器人有用任务行为的方法。