Deformable convolution, originally proposed for the adaptation to geometric
variations of objects, has recently shown compelling performance in aligning
multiple frames and is increasingly adopted for video super-resolution. Despite
its remarkable performance, its underlying mechanism for alignment remains
unclear. In this study, we carefully investigate the relation between
deformable alignment and the classic flow-based alignment. We show that
deformable convolution can be decomposed into a combination of spatial warping
and convolution. This decomposition reveals the commonality of deformable
alignment and flow-based alignment in formulation, but with a key difference in
their offset diversity. We further demonstrate through experiments that the
increased diversity in deformable alignment yields better-aligned features, and
hence significantly improves the quality of video super-resolution output.
Based on our observations, we propose an offset-fidelity loss that guides the
offset learning with optical flow. Experiments show that our loss successfully
avoids the overflow of offsets and alleviates the instability problem of
deformable alignment. Aside from the contributions to deformable alignment, our
formulation inspires a more flexible approach to introduce offset diversity to
flow-based alignment, improving its performance.

本研究对变形卷积与经典基于光流的卷积之间的联系进行了探究，发现变形卷积可以分解为空间变形和卷积的组合，具有与基于光流的卷积相似的表达式，但它们在偏移多样性方面有区别。实验证明，增加变形对齐的多样性可以获得更好的结果，同时提出一种偏移保真度损失来指导光流学习的偏移，在提高超分辨率视频质量的同时解决变形对齐的不稳定性问题，并启示了更灵活的引入偏移多样性来提高基于光流的对齐性能的途径。