Video compression performance is closely related to the accuracy of inter
prediction. It tends to be difficult to obtain accurate inter prediction for
the local video regions with inconsistent motion and occlusion. Traditional
video coding standards propose various technologies to handle motion
inconsistency and occlusion, such as recursive partitions, geometric
partitions, and long-term references. However, existing learned video
compression schemes focus on obtaining an overall minimized prediction error
averaged over all regions while ignoring the motion inconsistency and occlusion
in local regions. In this paper, we propose a spatial decomposition and
temporal fusion based inter prediction for learned video compression. To handle
motion inconsistency, we propose to decompose the video into structure and
detail (SDD) components first. Then we perform SDD-based motion estimation and
SDD-based temporal context mining for the structure and detail components to
generate short-term temporal contexts. To handle occlusion, we propose to
propagate long-term temporal contexts by recurrently accumulating the temporal
information of each historical reference feature and fuse them with short-term
temporal contexts. With the SDD-based motion model and long short-term temporal
contexts fusion, our proposed learned video codec can obtain more accurate
inter prediction. Comprehensive experimental results demonstrate that our codec
outperforms the reference software of H.266/VVC on all common test datasets for
both PSNR and MS-SSIM.

本文提出了基于空间分解和时间融合的学习视频压缩的帧间预测方法，通过处理运动不一致性和遮挡现象，结合长短时时间上下文融合，提高了视频压缩的帧间预测准确性。实验证明，所提出的编码器在所有常见测试数据集上的 PSNR 和 MS-SSIM 上优于 H.266 / VVC 的参考软件。

基于空间分解和时间融合的基于学习的视频压缩的帧间预测

Spatial Decomposition and Temporal Fusion based Inter Prediction for  Learned Video Compression

Inter prediction is an important module in video coding for temporal
redundancy removal, where similar reference blocks are searched from previously
coded frames and employed to predict the block to be coded. Although
traditional video codecs can estimate and compensate for block-level motions,
their inter prediction performance is still heavily affected by the remaining
inconsistent pixel-wise displacement caused by irregular rotation and
deformation. In this paper, we address the problem by proposing a deep frame
interpolation network to generate additional reference frames in coding
scenarios. First, we summarize the previous adaptive convolutions used for
frame interpolation and propose a factorized kernel convolutional network to
improve the modeling capacity and simultaneously keep its compact form. Second,
to better train this network, multi-domain hierarchical constraints are
introduced to regularize the training of our factorized kernel convolutional
network. For spatial domain, we use a gradually down-sampled and up-sampled
auto-encoder to generate the factorized kernels for frame interpolation at
different scales. For quality domain, considering the inconsistent quality of
the input frames, the factorized kernel convolution is modulated with
quality-related features to learn to exploit more information from high quality
frames. For frequency domain, a sum of absolute transformed difference loss
that performs frequency transformation is utilized to facilitate network
optimization from the view of coding performance. With the well-designed frame
interpolation network regularized by multi-domain hierarchical constraints, our
method surpasses HEVC on average 6.1% BD-rate saving and up to 11.0% BD-rate
saving for the luma component under the random access configuration.

本文提出了一种深度帧内插网络，通过多域分层约束来优化其训练并生成额外的参考帧，从而在编码中去除视频中像素级位移的影响，实现在随机存取配置下比 HEVC 平均 6.1％和最多 11.0％的 BD-rate 节省。