Though action recognition in videos has achieved great success recently, it
remains a challenging task due to the massive computational cost. Designing
lightweight networks is a possible solution, but it may degrade the recognition
performance. In this paper, we innovatively propose a general dynamic inference
idea to improve inference efficiency by leveraging the variation in the
distinguishability of different videos. The dynamic inference approach can be
achieved from aspects of the network depth and the number of input video
frames, or even in a joint input-wise and network depth-wise manner. In a
nutshell, we treat input frames and network depth of the computational graph as
a 2-dimensional grid, and several checkpoints are placed on this grid in
advance with a prediction module. The inference is carried out progressively on
the grid by following some predefined route, whenever the inference process
comes across a checkpoint, an early prediction can be made depending on whether
the early stop criteria meets. For the proof-of-concept purpose, we instantiate
three dynamic inference frameworks using two well-known backbone CNNs. In these
instances, we overcome the drawback of limited temporal coverage resulted from
an early prediction by a novel frame permutation scheme, and alleviate the
conflict between progressive computation and video temporal relation modeling
by introducing an online temporal shift module. Extensive experiments are
conducted to thoroughly analyze the effectiveness of our ideas and to inspire
future research efforts. Results on various datasets also evident the
superiority of our approach.

本文提出了一种动态推断方法，从网络深度和输入视频帧数的角度来提高推断效率，通过在输入帧和计算图的深度之间建立将预测模块放置在预先选择的检查点上，并按照预定义的路径逐步在网格上进行推断，可在中途进行预测和提前停止。通过实例化三个动态推断框架并使用两个知名的骨干卷积神经网络，证明了新方法的优越性。