Text-Video Retrieval (TVR) aims to align relevant video content with natural
language queries. To date, most state-of-the-art TVR methods learn
image-to-video transfer learning based on large-scale pre-trained
visionlanguage models (e.g., CLIP). However, fully fine-tuning these
pre-trained models for TVR incurs prohibitively expensive computation costs. To
this end, we propose to conduct efficient text-video Retrieval with a
sparse-andcorrelated AdaPter (RAP), i.e., fine-tuning the pre-trained model
with a few parameterized layers. To accommodate the text-video scenario, we
equip our RAP with two indispensable characteristics: temporal sparsity and
correlation. Specifically, we propose a low-rank modulation module to refine
the per-image features from the frozen CLIP backbone, which accentuates salient
frames within the video features while alleviating temporal redundancy.
Besides, we introduce an asynchronous self-attention mechanism that first
selects the top responsive visual patches and augments the correlation modeling
between them with learnable temporal and patch offsets. Extensive experiments
on four TVR datasets demonstrate that RAP achieves superior or comparable
performance compared to the fully fine-tuned counterpart and other
parameter-efficient fine-tuning methods.

使用一种称为 RAP 的稀疏且相关的 AdaPter 模型，通过在少数参数化层上进行预训练模型的微调，以提供高效的文本视频检索。RAP 配备了两个必要特征：时间稀疏性和相关性建模。通过引入低秩调制模块和异步自注意力机制，RAP 能够在四个文本视频检索数据集上表现出优越或可比较的性能。

基于稀疏相关适配器的高效文本 - 视频检索

RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adapter

Recurrent Neural Networks (RNNs) are useful in temporal sequence tasks.
However, training RNNs involves dense matrix multiplications which require
hardware that can support a large number of arithmetic operations and memory
accesses. Implementing online training of RNNs on the edge calls for optimized
algorithms for an efficient deployment on hardware. Inspired by the spiking
neuron model, the Delta RNN exploits temporal sparsity during inference by
skipping over the update of hidden states from those inactivated neurons whose
change of activation across two timesteps is below a defined threshold. This
work describes a training algorithm for Delta RNNs that exploits temporal
sparsity in the backward propagation phase to reduce computational requirements
for training on the edge. Due to the symmetric computation graphs of forward
and backward propagation during training, the gradient computation of
inactivated neurons can be skipped. Results show a reduction of $\sim$80% in
matrix operations for training a 56k parameter Delta LSTM on the Fluent Speech
Commands dataset with negligible accuracy loss. Logic simulations of a hardware
accelerator designed for the training algorithm show 2-10X speedup in matrix
computations for an activation sparsity range of 50%-90%. Additionally, we show
that the proposed Delta RNN training will be useful for online incremental
learning on edge devices with limited computing resources.

Delta RNN 利用时间稀疏性在推理过程中跳过被禁用神经元的隐藏状态更新，其训练算法在反向传播阶段利用时间稀疏性来减少边缘计算的计算要求。