Video super-resolution (VSR) aims to restore a sequence of high-resolution
(HR) frames from their low-resolution (LR) counterparts. Although some progress
has been made, there are grand challenges to effectively utilize temporal
dependency in entire video sequences. Existing approaches usually align and
aggregate video frames from limited adjacent frames (e.g., 5 or 7 frames),
which prevents these approaches from satisfactory results. In this paper, we
take one step further to enable effective spatio-temporal learning in videos.
We propose a novel Trajectory-aware Transformer for Video Super-Resolution
(TTVSR). In particular, we formulate video frames into several pre-aligned
trajectories which consist of continuous visual tokens. For a query token,
self-attention is only learned on relevant visual tokens along spatio-temporal
trajectories. Compared with vanilla vision Transformers, such a design
significantly reduces the computational cost and enables Transformers to model
long-range features. We further propose a cross-scale feature tokenization
module to overcome scale-changing problems that often occur in long-range
videos. Experimental results demonstrate the superiority of the proposed TTVSR
over state-of-the-art models, by extensive quantitative and qualitative
evaluations in four widely-used video super-resolution benchmarks. Both code
and pre-trained models can be downloaded at
this https URL

本研究提出了一种基于轨迹感知 Transformer 的视频超分辨率方法，利用空间和时间的信息来实现视频帧的超分辨率，实验证明该方法优于现有的超分辨率模型。

学习轨迹感知的视频超分辨率转换器

Learning Trajectory-Aware Transformer for Video Super-Resolution

Future prediction, especially in long-range videos, requires reasoning from
current and past observations. In this work, we address questions of temporal
extent, scaling, and level of semantic abstraction with a flexible
multi-granular temporal aggregation framework. We show that it is possible to
achieve state of the art in both next action and dense anticipation with simple
techniques such as max-pooling and attention. To demonstrate the anticipation
capabilities of our model, we conduct experiments on Breakfast, 50Salads, and
EPIC-Kitchens datasets, where we achieve state-of-the-art results. With minimal
modifications, our model can also be extended for video segmentation and action
recognition.

本文提出了一种灵活的多粒度时间聚合框架，用简单的技术如最大池化和注意力实现了最新的下一步行动和密集预测，并在 Breakfast，50Salads 和 EPIC-Kitchens 数据集上取得了最新的实验结果，且兼容于视频分割和动作识别。