Video compression performance is closely related to the accuracy of inter
prediction. It tends to be difficult to obtain accurate inter prediction for
the local video regions with inconsistent motion and occlusion. Traditional
video coding standards propose various technologies to handle motion
inconsistency and occlusion, such as recursive partitions, geometric
partitions, and long-term references. However, existing learned video
compression schemes focus on obtaining an overall minimized prediction error
averaged over all regions while ignoring the motion inconsistency and occlusion
in local regions. In this paper, we propose a spatial decomposition and
temporal fusion based inter prediction for learned video compression. To handle
motion inconsistency, we propose to decompose the video into structure and
detail (SDD) components first. Then we perform SDD-based motion estimation and
SDD-based temporal context mining for the structure and detail components to
generate short-term temporal contexts. To handle occlusion, we propose to
propagate long-term temporal contexts by recurrently accumulating the temporal
information of each historical reference feature and fuse them with short-term
temporal contexts. With the SDD-based motion model and long short-term temporal
contexts fusion, our proposed learned video codec can obtain more accurate
inter prediction. Comprehensive experimental results demonstrate that our codec
outperforms the reference software of H.266/VVC on all common test datasets for
both PSNR and MS-SSIM.

本文提出了基于空间分解和时间融合的学习视频压缩的帧间预测方法，通过处理运动不一致性和遮挡现象，结合长短时时间上下文融合，提高了视频压缩的帧间预测准确性。实验证明，所提出的编码器在所有常见测试数据集上的 PSNR 和 MS-SSIM 上优于 H.266 / VVC 的参考软件。

基于空间分解和时间融合的基于学习的视频压缩的帧间预测

Spatial Decomposition and Temporal Fusion based Inter Prediction for  Learned Video Compression

Video coding has traditionally been developed to support services such as
video streaming, videoconferencing, digital TV, and so on. The main intent was
to enable human viewing of the encoded content. However, with the advances in
deep neural networks (DNNs), encoded video is increasingly being used for
automatic video analytics performed by machines. In applications such as
automatic traffic monitoring, analytics such as vehicle detection, tracking and
counting, would run continuously, while human viewing could be required
occasionally to review potential incidents. To support such applications, a new
paradigm for video coding is needed that will facilitate efficient
representation and compression of video for both machine and human use in a
scalable manner. In this manuscript, we introduce the first end-to-end
learnable video codec that supports a machine vision task in its base layer,
while its enhancement layer supports input reconstruction for human viewing.
The proposed system is constructed based on the concept of conditional coding
to achieve better compression gains. Comprehensive experimental evaluations
conducted on four standard video datasets demonstrate that our framework
outperforms both state-of-the-art learned and conventional video codecs in its
base layer, while maintaining comparable performance on the human vision task
in its enhancement layer. We will provide the implementation of the proposed
system at www.github.com upon completion of the review process.

我们提出了一种全新的视频编码系统，基于条件编码的概念构建，其基础层支持机器视觉任务，而其增强层支持人类视觉重建，实验证明在基础层上，我们的框架优于现有的学习型视频编码和传统视频编码，并且在增强层上有可比性的性能。