The alignment of adjacent frames is considered an essential operation in
video super-resolution (VSR). Advanced VSR models, including the latest VSR
Transformers, are generally equipped with well-designed alignment modules.
However, the progress of the self-attention mechanism may violate this common
sense. In this paper, we rethink the role of alignment in VSR Transformers and
make several counter-intuitive observations. Our experiments show that: (i) VSR
Transformers can directly utilize multi-frame information from unaligned
videos, and (ii) existing alignment methods are sometimes harmful to VSR
Transformers. These observations indicate that we can further improve the
performance of VSR Transformers simply by removing the alignment module and
adopting a larger attention window. Nevertheless, such designs will
dramatically increase the computational burden, and cannot deal with large
motions. Therefore, we propose a new and efficient alignment method called
patch alignment, which aligns image patches instead of pixels. VSR Transformers
equipped with patch alignment could demonstrate state-of-the-art performance on
multiple benchmarks. Our work provides valuable insights on how multi-frame
information is used in VSR and how to select alignment methods for different
networks/datasets. Codes and models will be released at
this https URL

本文重新思考了 VSR Transformers 中对齐模块的作用，并通过实验证明，采用 patch alignment 对 VSR Transformers 的性能有助于提高。