Due to spatial redundancy in remote sensing images, sparse tokens containing
rich information are usually involved in self-attention (SA) to reduce the
overall token numbers within the calculation, avoiding the high computational
cost issue in Vision Transformers. However, such methods usually obtain sparse
tokens by hand-crafted or parallel-unfriendly designs, posing a challenge to
reach a better balance between efficiency and performance. Different from them,
this paper proposes to use learnable meta tokens to formulate sparse tokens,
which effectively learn key information meanwhile improving the inference
speed. Technically, the meta tokens are first initialized from image tokens via
cross-attention. Then, we propose Dual Cross-Attention (DCA) to promote
information exchange between image tokens and meta tokens, where they serve as
query and key (value) tokens alternatively in a dual-branch structure,
significantly reducing the computational complexity compared to self-attention.
By employing DCA in the early stages with dense visual tokens, we obtain the
hierarchical architecture LeMeViT with various sizes. Experimental results in
classification and dense prediction tasks show that LeMeViT has a significant
$1.7 \times$ speedup, fewer parameters, and competitive performance compared to
the baseline models, and achieves a better trade-off between efficiency and
performance.

本文提出使用可学习的元标记来构建稀疏标记，通过交叉注意力和双重交叉注意力（DCA）实现信息交换，从而达到提高图像识别速度和效果的目标。