Masked autoencoders (MAEs) have established themselves as a powerful method
for unsupervised pre-training for computer vision tasks. While vanilla MAEs put
equal emphasis on reconstructing the individual parts of the image, we propose
to inform the reconstruction process through an attention-guided loss function.
By leveraging advances in unsupervised object discovery, we obtain an attention
map of the scene which we employ in the loss function to put increased emphasis
on reconstructing relevant objects, thus effectively incentivizing the model to
learn more object-focused representations without compromising the established
masking strategy. Our evaluations show that our pre-trained models learn better
latent representations than the vanilla MAE, demonstrated by improved linear
probing and k-NN classification results on several benchmarks while at the same
time making ViTs more robust against varying backgrounds.

利用注重对象的重建过程来指导复原能力的建议，通过在损失函数中利用场景的注意力图获取的注意力图，提供更多的重建相关对象的强调，从而激励模型学习更加注重对象的表示，同时通过改进的线性探测和 k-NN 分类在几个基准测试中展示出我们预训练模型具有更好的潜在表示能力，同时使 ViTs 对不同背景更加稳健。