Event cameras sense the intensity changes asynchronously and produce event
streams with high dynamic range and low latency. This has inspired research
endeavors utilizing events to guide the challenging video superresolution (VSR)
task. In this paper, we make the first attempt to address a novel problem of
achieving VSR at random scales by taking advantages of the high temporal
resolution property of events. This is hampered by the difficulties of
representing the spatial-temporal information of events when guiding VSR. To
this end, we propose a novel framework that incorporates the spatial-temporal
interpolation of events to VSR in a unified framework. Our key idea is to learn
implicit neural representations from queried spatial-temporal coordinates and
features from both RGB frames and events. Our method contains three parts.
Specifically, the Spatial-Temporal Fusion (STF) module first learns the 3D
features from events and RGB frames. Then, the Temporal Filter (TF) module
unlocks more explicit motion information from the events near the queried
timestamp and generates the 2D features. Lastly, the SpatialTemporal Implicit
Representation (STIR) module recovers the SR frame in arbitrary resolutions
from the outputs of these two modules. In addition, we collect a real-world
dataset with spatially aligned events and RGB frames. Extensive experiments
show that our method significantly surpasses the prior-arts and achieves VSR
with random scales, e.g., 6.5. Code and dataset are available at https:
//vlis2022.this http URL.

本文提出一种新的方法，利用事件的高时空分辨率特性通过空时插值将事件引导具有随机比例因子的视频超分辨率任务。利用空时融合模块、时间滤波模块和空时隐式表示模块将 RGB 帧与事件的特征图结合来完成超分辨率恢复，实验结果表明，该方法显著超过以往技术。