We present a large empirical investigation on the use of pre-trained visual
representations (PVRs) for training downstream policies that execute real-world
tasks. Our study spans five different PVRs, two different policy-learning
paradigms (imitation and reinforcement learning), and three different robots
for 5 distinct manipulation and indoor navigation tasks. From this effort, we
can arrive at three insights: 1) the performance trends of PVRs in the
simulation are generally indicative of their trends in the real world, 2) the
use of PVRs enables a first-of-its-kind result with indoor ImageNav (zero-shot
transfer to a held-out scene in the real world), and 3) the benefits from
variations in PVRs, primarily data-augmentation and fine-tuning, also transfer
to the real-world performance. See project website for additional details and
visuals.

大规模实证研究表明预训练视觉表示（PVRs）的使用对训练执行现实任务的下游策略非常有用，尤其在操作和室内导航任务中表现出明显的性能优势。

大规模预训练视觉表示在模拟和真实环境中的研究：我们从中学到了什么？

What do we learn from a large-scale study of pre-trained visual  representations in sim and real environments?

We present the largest and most comprehensive empirical study of pre-trained
visual representations (PVRs) or visual 'foundation models' for Embodied AI.
First, we curate CortexBench, consisting of 17 different tasks spanning
locomotion, navigation, dexterous, and mobile manipulation. Next, we
systematically evaluate existing PVRs and find that none are universally
dominant.
To study the effect of pre-training data scale and diversity, we combine over
4,000 hours of egocentric videos from 7 different sources (over 5.6M images)
and ImageNet to train different-sized vision transformers using Masked
Auto-Encoding (MAE) on slices of this data. Contrary to inferences from prior
work, we find that scaling dataset size and diversity does not improve
performance universally (but does so on average).
Our largest model, named VC-1, outperforms all prior PVRs on average but does
not universally dominate either. Finally, we show that task or domain-specific
adaptation of VC-1 leads to substantial gains, with VC-1 (adapted) achieving
competitive or superior performance than the best known results on all of the
benchmarks in CortexBench. These models required over 10,000 GPU-hours to train
and can be found on our website for the benefit of the research community.

该研究是关于预先训练视觉表示（PVRs）或视觉基础模型在具体应用中的最全面的实证研究。研究发现，尽管数据规模和多样性通常能提高性能，但是没有任何一种 PVR 可以普遍占优。然而，通过对 VC-1 进行特定任务适应性的调整，可在 CortexBench 的所有基准测试中实现与最佳结果相当或更好的性能表现。