Masked visual modeling has attracted much attention due to its promising
potential in learning generalizable representations. Typical approaches urge
models to predict specific contents of masked tokens, which can be intuitively
considered as teaching a student (the model) to solve given problems
(predicting masked contents). Under such settings, the performance is highly
correlated with mask strategies (the difficulty of provided problems). We argue
that it is equally important for the model to stand in the shoes of a teacher
to produce challenging problems by itself. Intuitively, patches with high
values of reconstruction loss can be regarded as hard samples, and masking
those hard patches naturally becomes a demanding reconstruction task. To
empower the model as a teacher, we propose Hard Patches Mining (HPM),
predicting patch-wise losses and subsequently determining where to mask.
Technically, we introduce an auxiliary loss predictor, which is trained with a
relative objective to prevent overfitting to exact loss values. Also, to
gradually guide the training procedure, we propose an easy-to-hard mask
strategy. Empirically, HPM brings significant improvements under both image and
video benchmarks. Interestingly, solely incorporating the extra loss prediction
objective leads to better representations, verifying the efficacy of
determining where is hard to reconstruct. The code is available at
this https URL

通过引入一个辅助损失预测器和一个逐渐引导训练过程的易难遮罩策略，我们提出了 Hard Patches Mining (HPM) 方法来增强模型作为教师的能力，通过预测基于补丁的损失并确定遮罩的位置，从而显著改进了图像和视频的基准性能，并验证了确定难以重建的位置对于更好的表示的有效性。