Masked visual modeling has attracted much attention due to its promising
potential in learning generalizable representations. Typical approaches urge
models to predict specific contents of masked tokens, which can be intuitively
considered as teaching a student (the model) to solve given problems
(predicting masked contents). Under such settings, the performance is highly
correlated with mask strategies (the difficulty of provided problems). We argue
that it is equally important for the model to stand in the shoes of a teacher
to produce challenging problems by itself. Intuitively, patches with high
values of reconstruction loss can be regarded as hard samples, and masking
those hard patches naturally becomes a demanding reconstruction task. To
empower the model as a teacher, we propose Hard Patches Mining (HPM),
predicting patch-wise losses and subsequently determining where to mask.
Technically, we introduce an auxiliary loss predictor, which is trained with a
relative objective to prevent overfitting to exact loss values. Also, to
gradually guide the training procedure, we propose an easy-to-hard mask
strategy. Empirically, HPM brings significant improvements under both image and
video benchmarks. Interestingly, solely incorporating the extra loss prediction
objective leads to better representations, verifying the efficacy of
determining where is hard to reconstruct. The code is available at
this https URL

通过引入一个辅助损失预测器和一个逐渐引导训练过程的易难遮罩策略，我们提出了 Hard Patches Mining (HPM) 方法来增强模型作为教师的能力，通过预测基于补丁的损失并确定遮罩的位置，从而显著改进了图像和视频的基准性能，并验证了确定难以重建的位置对于更好的表示的有效性。

通过硬补丁挖掘引导掩蔽视觉建模

Bootstrap Masked Visual Modeling via Hard Patches Mining

Masked image modeling (MIM) has attracted much research attention due to its
promising potential for learning scalable visual representations. In typical
approaches, models usually focus on predicting specific contents of masked
patches, and their performances are highly related to pre-defined mask
strategies. Intuitively, this procedure can be considered as training a student
(the model) on solving given problems (predict masked patches). However, we
argue that the model should not only focus on solving given problems, but also
stand in the shoes of a teacher to produce a more challenging problem by
itself. To this end, we propose Hard Patches Mining (HPM), a brand-new
framework for MIM pre-training. We observe that the reconstruction loss can
naturally be the metric of the difficulty of the pre-training task. Therefore,
we introduce an auxiliary loss predictor, predicting patch-wise losses first
and deciding where to mask next. It adopts a relative relationship learning
strategy to prevent overfitting to exact reconstruction loss values.
Experiments under various settings demonstrate the effectiveness of HPM in
constructing masked images. Furthermore, we empirically find that solely
introducing the loss prediction objective leads to powerful representations,
verifying the efficacy of the ability to be aware of where is hard to
reconstruct.

本文提出了一个基于 Hard Patches Mining (HPM) 的框架，用于 Masked Image Modeling (MIM) 的预训练，采用相对关系学习策略，通过预测损失值来决定下一步的掩码。实验证明仅引入损失预测目标就能产生有效的表征，HMP 有效地构建了掩码图像，能够自我产生更具挑战性的问题。