Optimizing video inference efficiency has become increasingly important with
the growing demand for video analysis in various fields. Some existing methods
achieve high efficiency by explicit discard of spatial or temporal information,
which poses challenges in fast-changing and fine-grained scenarios. To address
these issues, we propose an efficient video representation network with
Differentiable Resolution Compression and Alignment mechanism, which compresses
non-essential information in the early stage of the network to reduce
computational costs while maintaining consistent temporal correlations.
Specifically, we leverage a Differentiable Context-aware Compression Module to
encode the saliency and non-saliency frame features, refining and updating the
features into a high-low resolution video sequence. To process the new
sequence, we introduce a new Resolution-Align Transformer Layer to capture
global temporal correlations among frame features with different resolutions,
while reducing spatial computation costs quadratically by utilizing fewer
spatial tokens in low-resolution non-saliency frames. The entire network can be
end-to-end optimized via the integration of the differentiable compression
module. Experimental results show that our method achieves the best trade-off
between efficiency and performance on near-duplicate video retrieval and
competitive results on dynamic video classification compared to
state-of-the-art methods. Code:this https URL

通过 Differentiable Resolution Compression and Alignment 机制，我们提出了一种高效的视频表示网络，以减小计算成本，同时保持一致的时间相关性，从而在近似重复视频检索和动态视频分类方面取得了最佳的效率与性能的权衡。