Bicubic downscaling is a prevalent technique used to reduce the video storage
burden or to accelerate the downstream processing speed. However, the inverse
upscaling step is non-trivial, and the downscaled video may also deteriorate
the performance of downstream tasks. In this paper, we propose a
self-conditioned probabilistic framework for video rescaling to learn the
paired downscaling and upscaling procedures simultaneously. During the
training, we decrease the entropy of the information lost in the downscaling by
maximizing its probability conditioned on the strong spatial-temporal prior
information within the downscaled video. After optimization, the downscaled
video by our framework preserves more meaningful information, which is
beneficial for both the upscaling step and the downstream tasks, e.g., video
action recognition task. We further extend the framework to a lossy video
compression system, in which a gradient estimator for non-differential
industrial lossy codecs is proposed for the end-to-end training of the whole
system. Extensive experimental results demonstrate the superiority of our
approach on video rescaling, video compression, and efficient action
recognition tasks.

本文提出了自条件概率框架的视频缩放方法，通过最大化概率来减少信息损失并保留更多有意义的信息，达到改善下游任务（如视频动作识别）、提高视频压缩效率等效果。该方法还进一步扩展到了有损视频压缩系统中，并且取得了在视频缩放、视频压缩和效率动作识别任务上的优越表现。

视频缩放的自条件概率学习

Self-Conditioned Probabilistic Learning of Video Rescaling

This paper addresses the video rescaling task, which arises from the needs of
adapting the video spatial resolution to suit individual viewing devices. We
aim to jointly optimize video downscaling and upscaling as a combined task.
Most recent studies focus on image-based solutions, which do not consider
temporal information. We present two joint optimization approaches based on
invertible neural networks with coupling layers. Our Long Short-Term Memory
Video Rescaling Network (LSTM-VRN) leverages temporal information in the
low-resolution video to form an explicit prediction of the missing
high-frequency information for upscaling. Our Multi-input Multi-output Video
Rescaling Network (MIMO-VRN) proposes a new strategy for downscaling and
upscaling a group of video frames simultaneously. Not only do they outperform
the image-based invertible model in terms of quantitative and qualitative
results, but also show much improved upscaling quality than the video rescaling
methods without joint optimization. To our best knowledge, this work is the
first attempt at the joint optimization of video downscaling and upscaling.

本文针对视频重缩放问题展开研究，提出了基于耦合层可逆神经网络的联合优化方法，LSTM-VRN 利用时间信息进行低分辨率视频上的高频信息显式预测，MIMO-VRN 提出了一种同时降低和提高视频帧组的新策略。与基于图像的可逆模型相比，这两种方法在定量和定性结果上都优于它们，同时也比没有联合优化的视频重缩放方法表现得更好。据我们所知，这是首次尝试联合优化视频降低和提高。