Convolutional neural networks have enabled accurate image super-resolution in
real-time. However, recent attempts to benefit from temporal correlations in
video super-resolution have been limited to naive or inefficient architectures.
In this paper, we introduce spatio-temporal sub-pixel convolution networks that
effectively exploit temporal redundancies and improve reconstruction accuracy
while maintaining real-time speed. Specifically, we discuss the use of early
fusion, slow fusion and 3D convolutions for the joint processing of multiple
consecutive video frames. We also propose a novel joint motion compensation and
video super-resolution algorithm that is orders of magnitude more efficient
than competing methods, relying on a fast multi-resolution spatial transformer
module that is end-to-end trainable. These contributions provide both higher
accuracy and temporally more consistent videos, which we confirm qualitatively
and quantitatively. Relative to single-frame models, spatio-temporal networks
can either reduce the computational cost by 30% whilst maintaining the same
quality or provide a 0.2dB gain for a similar computational cost. Results on
publicly available datasets demonstrate that the proposed algorithms surpass
current state-of-the-art performance in both accuracy and efficiency.

本论文提出一种新的视频超分辨率算法，即同时处理多个连续视频帧的 spatio-temporal sub-pixel convolution networks，并采用了创新性的运动补偿和视频超分辨率算法，相比于单帧模型，网络可以减少 30％的计算成本同时保持相同的质量，或者在相似的计算成本下提供 0.2dB 的增益。在公开数据集上的结果表明，该算法在精度和效率方面均超过当前的最新性能。