Masked Image Modeling (MIM) arises as a promising option for Vision
Transformers among various self-supervised learning (SSL) methods. The essence
of MIM lies in token-wise masked patch predictions, with targets patchified
from images; or generated by pre-trained tokenizers or models. We argue targets
from the pre-trained models usually exhibit spatial inconsistency, which makes
it excessively challenging for the model to follow to learn more discriminative
representations. To mitigate the issue, we introduce a novel self-supervision
signal based on Dynamic Token Morphing (DTM), which dynamically aggregates
contextually related tokens. DTM can be generally applied to various SSL
frameworks, yet we propose a simple MIM that employs DTM to effectively improve
the performance barely introducing extra training costs. Our experiments on
ImageNet-1K and ADE20K evidently demonstrate the superiority of our methods.
Furthermore, the comparative evaluation of iNaturalist and Fine-grained Visual
Classification datasets further validates the transferability of our method on
various downstream tasks. Our code will be released publicly.

基于动态令牌变形的掩码图像建模（DTM）是一种有前景的自监督学习方法，可应用于不同的 SSL 框架，在不引入额外的训练成本的情况下有效改进表现，并经过多个实验证明其在各种下游任务上的可迁移性。