Recent research has revealed that reducing the temporal and spatial
redundancy are both effective approaches towards efficient video recognition,
e.g., allocating the majority of computation to a task-relevant subset of
frames or the most valuable image regions of each frame. However, in most
existing works, either type of redundancy is typically modeled with another
absent. This paper explores the unified formulation of spatial-temporal dynamic
computation on top of the recently proposed AdaFocusV2 algorithm, contributing
to an improved AdaFocusV3 framework. Our method reduces the computational cost
by activating the expensive high-capacity network only on some small but
informative 3D video cubes. These cubes are cropped from the space formed by
frame height, width, and video duration, while their locations are adaptively
determined with a light-weighted policy network on a per-sample basis. At test
time, the number of the cubes corresponding to each video is dynamically
configured, i.e., video cubes are processed sequentially until a sufficiently
reliable prediction is produced. Notably, AdaFocusV3 can be effectively trained
by approximating the non-differentiable cropping operation with the
interpolation of deep features. Extensive empirical results on six benchmark
datasets (i.e., ActivityNet, FCVID, Mini-Kinetics, Something-Something V1&V2
and Diving48) demonstrate that our model is considerably more efficient than
competitive baselines.

本文探索在改进的 AdaFocusV3 框架上，统一时空动态计算的形式，通过在一些信息丰富的三维视频块上激活高性能网络以降低计算成本，并通过自适应轻量的策略网络在每个样本上根据测试时间的需求动态配置视频块数量，模型经过在 ActivityNet，FCVID，Mini-Kinetics，Something-Something V1&V2 和 Diving48 六个基准数据集上的广泛实验结果证明其显著比竞争算法更有效率。