We present Masked Frequency Modeling (MFM), a unified frequency-domain-based
approach for self-supervised pre-training of visual models. Instead of randomly
inserting mask tokens to the input embeddings in the spatial domain, in this
paper, we shift the perspective to the frequency domain. Specifically, MFM
first masks out a portion of frequency components of the input image and then
predicts the missing frequencies on the frequency spectrum. Our key insight is
that predicting masked components in the frequency domain is more ideal to
reveal underlying image patterns rather than predicting masked patches in the
spatial domain, due to the heavy spatial redundancy. Our findings suggest that
with the right configuration of mask-and-predict strategy, both the structural
information within high-frequency components and the low-level statistics among
low-frequency counterparts are useful in learning good representations. For the
first time, MFM demonstrates that, for both ViT and CNN, a simple non-Siamese
framework can learn meaningful representations even using none of the
following: (i) extra data, (ii) extra model, (iii) mask token. Experimental
results on image classification and semantic segmentation, as well as several
robustness benchmarks show the competitive performance and advanced robustness
of MFM compared with recent masked image modeling approaches. Furthermore, we
also comprehensively investigate the effectiveness of classical image
restoration tasks for representation learning from a unified frequency
perspective and reveal their intriguing relations with our MFM approach.

本文提出了一种基于频域的自监督预训练的方法，称作 Masked Frequency Modeling（MFM），通过在图像的频率分量上进行遮蔽，并预测其丢失的频率信息来学习图像的表示，实验证明该方法在图像分类、语义分割以及鲁棒性测试方面的性能优于其他最近的基于遮蔽的图像建模方法。此外，本文还全面调查了传统图像恢复任务对于从频域角度进行表示学习的有效性，并揭示了它们与 MFM 方法之间的有趣关系。