Vision Transformers (ViTs) have become increasingly popular in large-scale
Vision and Language Pre-training (VLP) models. Although previous VLP research
has demonstrated the efficacy of ViTs, these efforts still struggle with
computational inefficiencies caused by lengthy visual sequences. To address
this challenge, we introduce an efficient VLP approach called TRIPS, which
stands for Text-Relevant Image Patch Selection. TRIPS progressively reduces the
visual sequence using a text-guided patch-selection layer in the visual
backbone, thereby accelerating both training and inference processes. This
patch-selection layer dynamically computes text-dependent visual attention,
enabling it to identify attentive image tokens with text guidance and fuse
inattentive ones in an end-to-end fashion. Importantly, TRIPS does not add any
extra parameters and generalizes to most ViT-based VLP models. We incorporate
TRIPS into three representative VLP models covering single-stream, dual-stream,
and generative paradigms, and conduct extensive experiments on five widely-used
multi-modal benchmark datasets. Our experimental results reveal that TRIPS
delivers a 40% speedup, while maintaining competitive or superior performance
on downstream tasks.

TRIPS 是一种高效的 VLP 方法，通过在视觉主干中引入一个文本导向的补丁选择层，以渐进的方式减少视觉序列，从而加速训练和推理过程，并通过动态计算文本相关的视觉注意力来识别注意力图像令牌，以无缝融合不注意的令牌。TRIPS 不添加额外参数且适用于大多数基于 ViT 的 VLP 模型。将 TRIPS 整合到三种代表性的 VLP 模型中，并在五个广泛使用的多模式基准数据集上进行了详尽的实验研究。实验结果表明，TRIPS 能够提供 40% 的加速效果，同时在下游任务上保持有竞争力或更好的性能。