In this paper, we introduce a novel visual representation learning which
relies on a handful of adaptively learned tokens, and which is applicable to
both image and video understanding tasks. Instead of relying on hand-designed
splitting strategies to obtain visual tokens and processing a large number of
densely sampled patches for attention, our approach learns to mine important
tokens in visual data. This results in efficiently and effectively finding a
few important visual tokens and enables modeling of pairwise attention between
such tokens, over a longer temporal horizon for videos, or the spatial content
in images. Our experiments demonstrate strong performance on several
challenging benchmarks for both image and video recognition tasks. Importantly,
due to our tokens being adaptive, we accomplish competitive results at
significantly reduced compute amount. We obtain comparable results to the
state-of-the-arts on ImageNet while being computationally more efficient. We
also confirm the effectiveness of the approach on multiple video datasets,
including Kinetics-400, Kinetics-600, Charades, and AViD.
The code is available at:
this https URL

本文介绍了一种新的视觉表示学习方法，它依赖于少量自适应学习的令牌，并适用于图像和视频理解任务。与依赖手动设计的分割策略和处理大量密集抽样补丁以获取注意力的方法不同，我们的方法学习从视觉数据中挖掘重要令牌，从而有效地找到一些重要的视觉令牌，并使其能够对视频中更长的时空范围或图像中的空间内容进行配对注意力建模，同时具有更高的计算效率。在多个具有挑战性的基准测试中表现强劲，并且获得了与 ImageNet 的最新结果可比的结果，同时计算量显著降低。我们还在多个视频数据集（包括 Kinetics-400，Kinetics-600，Charades 和 AViD）上验证了该方法的有效性。