The main challenge in learning image-conditioned robotic policies is
acquiring a visual representation conducive to low-level control. Due to the
high dimensionality of the image space, learning a good visual representation
requires a considerable amount of visual data. However, when learning in the
real world, data is expensive. Sim2Real is a promising paradigm for overcoming
data scarcity in the real-world target domain by using a simulator to collect
large amounts of cheap data closely related to the target task. However, it is
difficult to transfer an image-conditioned policy from sim to real when the
domains are very visually dissimilar. To bridge the sim2real visual gap, we
propose using natural language descriptions of images as a unifying signal
across domains that captures the underlying task-relevant semantics. Our key
insight is that if two image observations from different domains are labeled
with similar language, the policy should predict similar action distributions
for both images. We demonstrate that training the image encoder to predict the
language description or the distance between descriptions of a sim or real
image serves as a useful, data-efficient pretraining step that helps learn a
domain-invariant image representation. We can then use this image encoder as
the backbone of an IL policy trained simultaneously on a large amount of
simulated and a handful of real demonstrations. Our approach outperforms widely
used prior sim2real methods and strong vision-language pretraining baselines
like CLIP and R3M by 25 to 40%.

通过使用自然语言描述图像作为统一的信号来捕捉底层的任务相关语义，我们提出了一种方法来解决在学习图像条件机器人策略时所面临的模拟与真实世界之间的视觉差距问题，该方法通过对大量模拟数据和少量真实演示进行同时训练，将图像编码器作为无域变迁的图像表示的基础，并取得了较之前的 Sim2Real 方法和 CLIP、R3M 等强大的视觉 - 语言预训练方法 25 至 40% 的性能提升。