In this paper, we introduce Attention Prompt Tuning (APT) - a computationally
efficient variant of prompt tuning for video-based applications such as action
recognition. Prompt tuning approaches involve injecting a set of learnable
prompts along with data tokens during fine-tuning while keeping the backbone
frozen. This approach greatly reduces the number of learnable parameters
compared to full tuning. For image-based downstream tasks, normally a couple of
learnable prompts achieve results close to those of full tuning. However,
videos, which contain more complex spatiotemporal information, require hundreds
of tunable prompts to achieve reasonably good results. This reduces the
parameter efficiency observed in images and significantly increases latency and
the number of floating-point operations (FLOPs) during inference. To tackle
these issues, we directly inject the prompts into the keys and values of the
non-local attention mechanism within the transformer block. Additionally, we
introduce a novel prompt reparameterization technique to make APT more robust
against hyperparameter selection. The proposed APT approach greatly reduces the
number of FLOPs and latency while achieving a significant performance boost
over the existing parameter-efficient tuning methods on UCF101, HMDB51, and
SSv2 datasets for action recognition. The code and pre-trained models are
available at this https URL

通过在 transformer 模块的键和值中直接插入提示信息，我们提出了注意力提示调整（Attention Prompt Tuning，APT）方法，从而在视频行为识别中大大减少了浮点运算和延迟，并在 UCF101、HMDB51 和 SSv2 数据集上实现了显著的性能提升。

注意力提示调整：参数高效的预训练模型适应时空建模

Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained  Models for Spatiotemporal Modeling

Recent years have seen remarkable progress in semantic segmentation. Yet, it
remains a challenging task to apply segmentation techniques to video-based
applications. Specifically, the high throughput of video streams, the sheer
cost of running fully convolutional networks, together with the low-latency
requirements in many real-world applications, e.g. autonomous driving, present
a significant challenge to the design of the video segmentation framework. To
tackle this combined challenge, we develop a framework for video semantic
segmentation, which incorporates two novel components: (1) a feature
propagation module that adaptively fuses features over time via spatially
variant convolution, thus reducing the cost of per-frame computation; and (2)
an adaptive scheduler that dynamically allocate computation based on accuracy
prediction. Both components work together to ensure low latency while
maintaining high segmentation quality. On both Cityscapes and CamVid, the
proposed framework obtained competitive performance compared to the state of
the art, while substantially reducing the latency, from 360 ms to 119 ms.

本研究提出了一种视频语义分割框架，该框架利用特征传播模块和自适应调度器相结合的方法，通过时间间隔自适应地融合特征并根据准确性预测动态分配计算量，以实现低延迟和高语义分割质量。实验结果表明，在与现有模型的比较中，该模型在 Cityscapes 和 CamVid 这两个数据集上获得了竞争性的性能，并将延迟从 360 ms 降低到 119 ms。