This paper represents a neat yet effective framework, named SemanticMIM, to
integrate the advantages of masked image modeling (MIM) and contrastive
learning (CL) for general visual representation. We conduct a thorough
comparative analysis between CL and MIM, revealing that their complementary
advantages fundamentally stem from two distinct phases, i.e., compression and
reconstruction. Specifically, SemanticMIM leverages a proxy architecture that
customizes interaction between image and mask tokens, bridging these two phases
to achieve general visual representation with the property of abundant semantic
and positional awareness. Through extensive qualitative and quantitative
evaluations, we demonstrate that SemanticMIM effectively amalgamates the
benefits of CL and MIM, leading to significant enhancement of performance and
feature linear separability. SemanticMIM also offers notable interpretability
through attention response visualization. Codes are available at
this https URL

本研究提出了一种简洁而有效的框架 SemanticMIM，以整合遮罩图像建模（MIM）和对比学习（CL）的优势，用于通用视觉表示。通过对 CL 和 MIM 进行彻底的比较分析，揭示了它们互补优势根源于压缩和重建两个不同阶段，而 SemanticMIM 利用代理架构自定义图像和掩码令牌之间的交互，以丰富的语义和位置感知性实现通用视觉表示。通过广泛的定性和定量评估，我们证明了 SemanticMIM 有效地融合了 CL 和 MIM 的优点，显著提升了性能和特征的线性可分性，并提供了引人注目的注意力响应可视化。