Position Embeddings (PEs), an arguably indispensable component in Vision
Transformers (ViTs), have been shown to improve the performance of ViTs on many
vision tasks. However, PEs have a potentially high risk of privacy leakage
since the spatial information of the input patches is exposed. This caveat
naturally raises a series of interesting questions about the impact of PEs on
the accuracy, privacy, prediction consistency, etc. To tackle these issues, we
propose a Masked Jigsaw Puzzle (MJP) position embedding method. In particular,
MJP first shuffles the selected patches via our block-wise random jigsaw puzzle
shuffle algorithm, and their corresponding PEs are occluded. Meanwhile, for the
non-occluded patches, the PEs remain the original ones but their spatial
relation is strengthened via our dense absolute localization regressor. The
experimental results reveal that 1) PEs explicitly encode the 2D spatial
relationship and lead to severe privacy leakage problems under gradient
inversion attack; 2) Training ViTs with the naively shuffled patches can
alleviate the problem, but it harms the accuracy; 3) Under a certain shuffle
ratio, the proposed MJP not only boosts the performance and robustness on
large-scale datasets (i.e., ImageNet-1K and ImageNet-C, -A/O) but also improves
the privacy preservation ability under typical gradient attacks by a large
margin. The source code and trained models are available
at~https://github.com/yhlleo/MJP.

提出一种名为 Masked Jigsaw Puzzle (MJP) 的位置嵌入方法，用于解决 ViT 中的位置嵌入对隐私泄露的问题。实验结果表明，MJP 方法在一定程度的遮蔽之下，既可以提高性能和鲁棒性，也可以显著提高在梯度攻击下的隐私保护能力。