How to properly model the inter-frame relation within the video sequence is
an important but unsolved challenge for video restoration (VR). In this work,
we propose an unsupervised flow-aligned sequence-to-sequence model (S2SVR) to
address this problem. On the one hand, the sequence-to-sequence model, which
has proven capable of sequence modeling in the field of natural language
processing, is explored for the first time in VR. Optimized serialization
modeling shows potential in capturing long-range dependencies among frames. On
the other hand, we equip the sequence-to-sequence model with an unsupervised
optical flow estimator to maximize its potential. The flow estimator is trained
with our proposed unsupervised distillation loss, which can alleviate the data
discrepancy and inaccurate degraded optical flow issues of previous flow-based
methods. With reliable optical flow, we can establish accurate correspondence
among multiple frames, narrowing the domain difference between 1D language and
2D misaligned frames and improving the potential of the sequence-to-sequence
model. S2SVR shows superior performance in multiple VR tasks, including video
deblurring, video super-resolution, and compressed video quality enhancement.
Code and models are publicly available at
this https URL

本文提出了一种无监督流对齐序列到序列模型（S2SVR），通过使用序列到序列模型和无监督光流估计器，并采用无监督蒸馏损失训练光流估计器，以解决视频恢复中间帧的建模问题。本文方法在多种视频恢复任务中均表现优异。