Visual representation learning hold great promise for robotics, but is
severely hampered by the scarcity and homogeneity of robotics datasets. Recent
works address this problem by pre-training visual representations on
large-scale but out-of-domain data (e.g., videos of egocentric interactions)
and then transferring them to target robotics tasks. While the field is heavily
focused on developing better pre-training algorithms, we find that dataset
choice is just as important to this paradigm's success. After all, the
representation can only learn the structures or priors present in the
pre-training dataset. To this end, we flip the focus on algorithms, and instead
conduct a dataset centric analysis of robotic pre-training. Our findings call
into question some common wisdom in the field. We observe that traditional
vision datasets (like ImageNet, Kinetics and 100 Days of Hands) are
surprisingly competitive options for visuo-motor representation learning, and
that the pre-training dataset's image distribution matters more than its size.
Finally, we show that common simulation benchmarks are not a reliable proxy for
real world performance and that simple regularization strategies can
dramatically improve real world policy learning.
this https URL

通过对机器人的预训练数据集进行集中分析，我们发现常规视觉数据集对于视觉 - 运动表示学习是竞争力强的选择，此外，预训练数据集的图像分布比其规模更为重要，同时简单的正规化策略可以显著改善真实世界策略学习。

关于视觉 - 动作预训练数据集的客观观察

An Unbiased Look at Datasets for Visuo-Motor Pre-Training

Probes are small networks that predict properties of underlying data from
embeddings, and they provide a targeted, effective way to illuminate the
information contained in embeddings. While analysis through the use of probes
has become standard in NLP, there has been much less exploration in vision.
Image foundation models have primarily been evaluated for semantic content.
Better understanding the non-semantic information in popular embeddings (e.g.,
MAE, SimCLR, or CLIP) will shed new light both on the training algorithms and
on the uses for these foundation models. We design a systematic transformation
prediction task and measure the visual content of embeddings along many axes,
including image style, quality, and a range of natural and artificial
transformations. Surprisingly, six embeddings (including SimCLR) encode enough
non-semantic information to identify dozens of transformations. We also
consider a generalization task, where we group similar transformations and hold
out several for testing. We find that image-text models (CLIP and ALIGN) are
better at recognizing new examples of style transfer than masking-based models
(CAN and MAE). Overall, our results suggest that the choice of pre-training
algorithm impacts the types of information in the embedding, and certain models
are better than others for non-semantic downstream tasks.

通过对多种视觉 embedding 进行系统转换任务的测试与比较，发现不同的网络预训练算法会在 embedding 中编码不同类型的图像非语义信息，并且这些信息可以被有效地利用到一些特定的下游应用中。

实质还是风格：你的图像嵌入知道什么？

Substance or Style: What Does Your Image Embedding Know?

Hierarchical neural architectures are often used to capture long-distance
dependencies and have been applied to many document-level tasks such as
summarization, document segmentation, and sentiment analysis. However,
effective usage of such a large context can be difficult to learn, especially
in the case where there is limited labeled data available. Building on the
recent success of language model pretraining methods for learning flat
representations of text, we propose algorithms for pre-training hierarchical
document representations from unlabeled data. Unlike prior work, which has
focused on pre-training contextual token representations or context-independent
{sentence/paragraph} representations, our hierarchical document representations
include fixed-length sentence/paragraph representations which integrate
contextual information from the entire documents. Experiments on document
segmentation, document-level question answering, and extractive document
summarization demonstrate the effectiveness of the proposed pre-training
algorithms.

从无标注数据中预训练了一种能够包括来自整个文档的上下文信息的分层文档表示，包括定长的句子 / 段落表示，并应用于文档分割、文档级问答和抽取式文档摘要等方面取得了有效结果。