Understanding whether self-supervised learning methods can scale with
unlimited data is crucial for training large-scale models. In this work, we
conduct an empirical study on the scaling capability of masked image modeling
(MIM) methods (e.g., MAE) for visual recognition. Unlike most previous works
that depend on the widely-used ImageNet dataset, which is manually curated and
object-centric, we take a step further and propose to investigate this problem
in a more practical setting. Specifically, we utilize the web-collected
Coyo-700M dataset. We randomly sample varying numbers of training images from
the Coyo dataset and construct a series of sub-datasets, containing 0.5M, 1M,
5M, 10M, and 100M images, for pre-training. Our goal is to investigate how the
performance changes on downstream tasks when scaling with different sizes of
data and models. The study reveals that: 1) MIM can be viewed as an effective
method to improve the model capacity when the scale of the training data is
relatively small; 2) Strong reconstruction targets can endow the models with
increased capacities on downstream tasks; 3) MIM pre-training is data-agnostic
under most scenarios, which means that the strategy of sampling pre-training
data is non-critical. We hope these observations could provide valuable
insights for future research on MIM.

本文旨在通过在使用 Coyo-700M 数据集进行实验，探究遮蔽图像建模 (MIM) 方法在不同大小和模型的情况下，进行下游任务的表现变化，并发现了 MIM 在训练数据规模较小时提高模型容量的效果、强重构目标能增加模型在下游任务中的容量，以及大多数情况下，MIM 预训练是数据不可知的。这些发现为未来的 MIM 研究提供了有价值的见解。

探究遮罩图像建模中的数据缩放

Delving Deeper into Data Scaling in Masked Image Modeling

This paper presents a scalable Bayesian technique for decentralized state
estimation from multiple platforms in dynamic environments. As has long been
recognized, centralized architectures impose severe scaling limitations for
distributed systems due to the enormous communication overheads. We propose a
strictly decentralized approach in which only nearby platforms exchange
information. They do so through an interactive communication protocol aimed at
maximizing information flow. Our approach is evaluated in the context of a
distributed surveillance scenario that arises in a robotic system for playing
the game of laser tag. Our results, both from simulation and using physical
robots, illustrate an unprecedented scaling capability to large teams of
vehicles.

本文介绍了一种可扩展的贝叶斯技术，用于在动态环境中从多个平台进行去中心化状态估计。通过严格分散的方法并通过互动通信协议促进信息流，实现仅在附近平台之间交换信息和在分布式监视方案的上下文下进行评估，其结果展示了对大型车队的前所未有的可扩展能力。