Temporal Action Localization (TAL) involves localizing and classifying action
snippets in an untrimmed video. The emergence of large video foundation models
has led RGB-only video backbones to outperform previous methods needing both
RGB and optical flow modalities. Leveraging these large models is often limited
to training only the TAL head due to the prohibitively large GPU memory
required to adapt the video backbone for TAL. To overcome this limitation, we
introduce LoSA, the first memory-and-parameter-efficient backbone adapter
designed specifically for TAL to handle untrimmed videos. LoSA specializes for
TAL by introducing Long-Short-range Adapters that adapt the intermediate layers
of the video backbone over different temporal ranges. These adapters run
parallel to the video backbone to significantly reduce memory footprint. LoSA
also includes Long-Short-range Fusion that strategically combines the output of
these adapters from the video backbone layers to enhance the video features
provided to the TAL head. Experiments show that LoSA significantly outperforms
all existing methods on standard TAL benchmarks, THUMOS-14 and
ActivityNet-v1.3, by scaling end-to-end backbone adaptation to
billion-parameter-plus models like VideoMAEv2~(ViT-g) and leveraging them
beyond head-only transfer learning.

通过引入第一个专门为处理未修剪视频的 TAL 设计的内存和参数高效的背骨适配器 LoSA，本研究在标准 TAL 基准测试 THUMOS-14 和 ActivityNet-v1.3 上明显优于现有方法，通过使视频骨干可以适应超过十亿参数的模型，如 VideoMAEv2（ViT-g），有效地扩展了端到端骨干适应的能力，并利用了这些模型进行非仅在头部的迁移学习。