3D convolutional networks are prevalent for video recognition. While
achieving excellent recognition performance on standard benchmarks, they
operate on a sequence of frames with 3D convolutions and thus are
computationally demanding. Exploiting large variations among different videos,
we introduce Ada3D, a conditional computation framework that learns
instance-specific 3D usage policies to determine frames and convolution layers
to be used in a 3D network. These policies are derived with a two-head
lightweight selection network conditioned on each input video clip. Then, only
frames and convolutions that are selected by the selection network are used in
the 3D model to generate predictions. The selection network is optimized with
policy gradient methods to maximize a reward that encourages making correct
predictions with limited computation. We conduct experiments on three video
recognition benchmarks and demonstrate that our method achieves similar
accuracies to state-of-the-art 3D models while requiring 20%-50% less
computation across different datasets. We also show that learned policies are
transferable and Ada3D is compatible to different backbones and modern clip
selection approaches. Our qualitative analysis indicates that our method
allocates fewer 3D convolutions and frames for "static" inputs, yet uses more
for motion-intensive clips.

通过 Ada3D 条件计算框架，使用策略梯度方法和轻量级选择网络，实现对不同视频的实例特定 3D 使用策略的学习，以决定使用哪些帧和卷积层，从而实现对 3D 模型进行预测，即使在不同数据集上，也能够实现类似于现有 3D 模型的准确度，且需要的计算量减少了 20％-50％。