We present the largest and most comprehensive empirical study of pre-trained
visual representations (PVRs) or visual 'foundation models' for Embodied AI.
First, we curate CortexBench, consisting of 17 different tasks spanning
locomotion, navigation, dexterous, and mobile manipulation. Next, we
systematically evaluate existing PVRs and find that none are universally
dominant.
To study the effect of pre-training data scale and diversity, we combine over
4,000 hours of egocentric videos from 7 different sources (over 5.6M images)
and ImageNet to train different-sized vision transformers using Masked
Auto-Encoding (MAE) on slices of this data. Contrary to inferences from prior
work, we find that scaling dataset size and diversity does not improve
performance universally (but does so on average).
Our largest model, named VC-1, outperforms all prior PVRs on average but does
not universally dominate either. Finally, we show that task or domain-specific
adaptation of VC-1 leads to substantial gains, with VC-1 (adapted) achieving
competitive or superior performance than the best known results on all of the
benchmarks in CortexBench. These models required over 10,000 GPU-hours to train
and can be found on our website for the benefit of the research community.

该研究是关于预先训练视觉表示（PVRs）或视觉基础模型在具体应用中的最全面的实证研究。研究发现，尽管数据规模和多样性通常能提高性能，但是没有任何一种 PVR 可以普遍占优。然而，通过对 VC-1 进行特定任务适应性的调整，可在 CortexBench 的所有基准测试中实现与最佳结果相当或更好的性能表现。

寻找一个为具身智能而设计的人工视皮层的进展

Where are we in the search for an Artificial Visual Cortex for Embodied  Intelligence?

Dense passage retrieval aims to retrieve the relevant passages of a query
from a large corpus based on dense representations (i.e., vectors) of the query
and the passages. Recent studies have explored improving pre-trained language
models to boost dense retrieval performance. This paper proposes CoT-MAE
(ConTextual Masked Auto-Encoder), a simple yet effective generative
pre-training method for dense passage retrieval. CoT-MAE employs an asymmetric
encoder-decoder architecture that learns to compress the sentence semantics
into a dense vector through self-supervised and context-supervised masked
auto-encoding. Precisely, self-supervised masked auto-encoding learns to model
the semantics of the tokens inside a text span, and context-supervised masked
auto-encoding learns to model the semantical correlation between the text
spans. We conduct experiments on large-scale passage retrieval benchmarks and
show considerable improvements over strong baselines, demonstrating the high
efficiency of CoT-MAE. Our code is available at
this https URL.

本研究提出 CoT-MAE 方法，通过自监督和上下文监督的遮蔽自编码学习将句子语义压缩成密集向量，进而在大规模的检索基准数据集上实现了相比强基线更高效的检索表现。

基于上下文遮盖的自编码器用于密集型段落检索

ConTextual Masked Auto-Encoder for Dense Passage Retrieval

Masked auto-encoding is a popular and effective self-supervised learning
approach to point cloud learning. However, most of the existing methods
reconstruct only the masked points and overlook the local geometry information,
which is also important to understand the point cloud data. In this work, we
make the first attempt, to the best of our knowledge, to consider the local
geometry information explicitly into the masked auto-encoding, and propose a
novel Masked Surfel Prediction (MaskSurf) method. Specifically, given the input
point cloud masked at a high ratio, we learn a transformer-based
encoder-decoder network to estimate the underlying masked surfels by
simultaneously predicting the surfel positions (i.e., points) and per-surfel
orientations (i.e., normals). The predictions of points and normals are
supervised by the Chamfer Distance and a newly introduced Position-Indexed
Normal Distance in a set-to-set manner. Our MaskSurf is validated on six
downstream tasks under three fine-tuning strategies. In particular, MaskSurf
outperforms its closest competitor, Point-MAE, by 1.2\% on the real-world
dataset of ScanObjectNN under the OBJ-BG setting, justifying the advantages of
masked surfel prediction over masked point cloud reconstruction. Codes will be
available at this https URL.

本研究提出一种名为 Masked Surfel Prediction 的方法，首次将局部几何信息显式考虑到遮罩自编码中，以预测替代遮罩部分的 surfels 的位置和朝向，并使用 Chamfer 距离和一种新的位置索引法来进行监督学习，在六项下游任务中验证了该方法的显著优越性。

自监督点云学习的遮盖表面预测

Masked Surfel Prediction for Self-Supervised Point Cloud Learning

Vision Transformers (ViT) become widely-adopted architectures for various
vision tasks. Masked auto-encoding for feature pretraining and multi-scale
hybrid convolution-transformer architectures can further unleash the potentials
of ViT, leading to state-of-the-art performances on image classification,
detection and semantic segmentation. In this paper, our ConvMAE framework
demonstrates that multi-scale hybrid convolution-transformer can learn more
discriminative representations via the mask auto-encoding scheme. However,
directly using the original masking strategy leads to the heavy computational
cost and pretraining-finetuning discrepancy. To tackle the issue, we adopt the
masked convolution to prevent information leakage in the convolution blocks. A
simple block-wise masking strategy is proposed to ensure computational
efficiency. We also propose to more directly supervise the multi-scale features
of the encoder to boost multi-scale features. Based on our pretrained ConvMAE
models, ConvMAE-Base improves ImageNet-1K finetuning accuracy by 1.4% compared
with MAE-Base. On object detection, ConvMAE-Base finetuned for only 25 epochs
surpasses MAE-Base fined-tuned for 100 epochs by 2.9% box AP and 2.2% mask AP
respectively. Code and pretrained models are available at
this https URL

本文介绍使用 ConvMAE 框架对 Vision Transformers 进行特征预训练和 Masked Auto-Encoder 技术的引入，提高了其在各种视觉任务中的表现。而使用 masked convolution 和直接监督卷积层的 features 等方法，可以在保证计算效率的同时提高了分类和检测的准确率。