In this paper, we propose a framework named OCSampler to explore a compact
yet effective video representation with one short clip for efficient video
recognition. Recent works prefer to formulate frame sampling as a sequential
decision task by selecting frames one by one according to their importance,
while we present a new paradigm of learning instance-specific video
condensation policies to select informative frames for representing the entire
video only in a single step. Our basic motivation is that the efficient video
recognition task lies in processing a whole sequence at once rather than
picking up frames sequentially. Accordingly, these policies are derived from a
light-weighted skim network together with a simple yet effective policy network
within one step. Moreover, we extend the proposed method with a frame number
budget, enabling the framework to produce correct predictions in high
confidence with as few frames as possible. Experiments on four benchmarks,
i.e., ActivityNet, Mini-Kinetics, FCVID, Mini-Sports1M, demonstrate the
effectiveness of our OCSampler over previous methods in terms of accuracy,
theoretical computational expense, actual inference speed. We also evaluate its
generalization power across different classifiers, sampled frames, and search
spaces. Especially, we achieve 76.9% mAP and 21.7 GFLOPs on ActivityNet with an
impressive throughput: 123.9 Videos/s on a single TITAN Xp GPU.

本文通过提出 OCSampler 框架，利用一小段短视频的紧凑且有效的表示来实现视频识别的高效率。利用实例特定的视频压缩策略和轻量的 skim 网络和简单且有效的策略网络，我们扩展了所提出的方法，具有帧数预算，可以在高置信度下用尽可能少的帧产生正确的预测。实验表明，该框架在准确性，理论计算开销和实际推理速度等方面优于以前的方法。