The joint-embedding predictive architecture (JEPA) recently has shown
impressive results in extracting visual representations from unlabeled imagery
under a masking strategy. However, we reveal its disadvantages, notably its
insufficient understanding of local semantics. This deficiency originates from
masked modeling in the embedding space, resulting in a reduction of
discriminative power and can even lead to the neglect of critical local
semantics. To bridge this gap, we introduce DMT-JEPA, a novel masked modeling
objective rooted in JEPA, specifically designed to generate discriminative
latent targets from neighboring information. Our key idea is simple: we
consider a set of semantically similar neighboring patches as a target of a
masked patch. To be specific, the proposed DMT-JEPA (a) computes feature
similarities between each masked patch and its corresponding neighboring
patches to select patches having semantically meaningful relations, and (b)
employs lightweight cross-attention heads to aggregate features of neighboring
patches as the masked targets. Consequently, DMT-JEPA demonstrates strong
discriminative power, offering benefits across a diverse spectrum of downstream
tasks. Through extensive experiments, we demonstrate our effectiveness across
various visual benchmarks, including ImageNet-1K image classification, ADE20K
semantic segmentation, and COCO object detection tasks. Code is available at:
https://github.com/DMTJEPA/DMTJEPA.

我们介绍了一种新的基于 JEPA 的 masked modeling 目标 DMT-JEPA，通过计算特征相似性和使用轻量级交叉注意力头，将语义相似的邻域图像作为目标来生成具有辨别力的潜在目标，从而弥补了 JEPA 在理解局部语义方面的不足。