Existing parameter-efficient fine-tuning (PEFT) methods have achieved
significant success on vision transformers (ViTs) adaptation by improving
parameter efficiency. However, the exploration of enhancing inference
efficiency during adaptation remains underexplored. This limits the broader
application of pre-trained ViT models, especially when the model is
computationally extensive. In this paper, we propose Dynamic Tuning (DyT), a
novel approach to improve both parameter and inference efficiency for ViT
adaptation. Specifically, besides using the lightweight adapter modules, we
propose a token dispatcher to distinguish informative tokens from less
important ones, allowing the latter to dynamically skip the original block,
thereby reducing the redundant computation during inference. Additionally, we
explore multiple design variants to find the best practice of DyT. Finally,
inspired by the mixture-of-experts (MoE) mechanism, we introduce an enhanced
adapter to further boost the adaptation performance. We validate DyT across
various tasks, including image/video recognition and semantic segmentation. For
instance, DyT achieves comparable or even superior performance compared to
existing PEFT methods while evoking only 71%-85% of their FLOPs on the VTAB-1K
benchmark.

现有的参数高效微调（PEFT）方法通过提高参数效率，在视觉变换器（ViTs）适应中取得了显著的成功，然而，在适应过程中改善推理效率的探索仍未充分开展，这限制了预训练 ViT 模型的广泛应用，特别是在计算上耗费较多的情况下。在本文中，我们提出了一种名为动态调整 (DyT) 的新方法，来改善 ViT 适应的参数和推理效率。具体而言，除了使用轻量级适配器模块外，我们还提出了一个标记分发器，用于区别信息丰富的标记和不重要的标记，从而使后者动态跳过原始块，减少推理过程中的冗余计算。此外，我们探索了多种设计变体以找到 DyT 的最佳实践。最后，受混合专家（MoE）机制的启发，我们引入了一个增强的适配器以进一步提高适应性能。我们在包括图像 / 视频识别和语义分割在内的各种任务中验证了 DyT。例如，在 VTAB-1K 基准测试中，DyT 在触发 71％-85％的 FLOPs 的情况下实现了与现有 PEFT 方法相当或甚至更优越的性能。