Temporal Video Frame Synthesis (TVFS) aims at synthesizing novel frames at
timestamps different from existing frames, which has wide applications in video
codec, editing and analysis. In this paper, we propose a high framerate TVFS
framework which takes hybrid input data from a low-speed frame-based sensor and
a high-speed event-based sensor. Compared to frame-based sensors, event-based
sensors report brightness changes at very high speed, which may well provide
useful spatio-temoral information for high framerate TVFS. In our framework, we
first introduce a differentiable forward model to approximate the physical
sensing process, fusing the two different modes of data as well as unifying a
variety of TVFS tasks, i.e., interpolation, prediction and motion deblur. We
leverage autodifferentiation which propagates the gradients of a loss defined
on the measured data back to the latent high framerate video. We show results
with better performance compared to state-of-the-art. Second, we develop a deep
learning-based strategy to enhance the results from the first step, which we
refer as a residual "denoising" process. Our trained "denoiser" is beyond
Gaussian denoising and shows properties such as contrast enhancement and motion
awareness. We show that our framework is capable of handling challenging scenes
including both fast motion and strong occlusions.

提出了一个高帧率的 TVFS 框架，该框架利用来自低速基于帧的传感器和高速事件传感器的混合输入数据，融合了两种不同的数据模式，并统一了各种 TVFS 任务，即插值、预测和运动去模糊，采用自动微分对梯度进行反向传播，实验结果表明与现有技术相比性能更佳。其次，采用深度学习策略增强了第一步结果，提出了一种残差 “去噪” 方法，展现了处理具有挑战性场景的能力，包括快速运动和强烈遮挡。

基于事件触发的视频帧合成

Event-driven Video Frame Synthesis

New vision sensors, such as the Dynamic and Active-pixel Vision sensor
(DAVIS), incorporate a conventional global-shutter camera and an event-based
sensor in the same pixel array. These sensors have great potential for
high-speed robotics and computer vision because they allow us to combine the
benefits of conventional cameras with those of event-based sensors: low
latency, high temporal resolution, and very high dynamic range. However, new
algorithms are required to exploit the sensor characteristics and cope with its
unconventional output, which consists of a stream of asynchronous brightness
changes (called "events") and synchronous grayscale frames. For this purpose,
we present and release a collection of datasets captured with a DAVIS in a
variety of synthetic and real environments, which we hope will motivate
research on new algorithms for high-speed and high-dynamic-range robotics and
computer-vision applications. In addition to global-shutter intensity images
and asynchronous events, we provide inertial measurements and ground-truth
camera poses from a motion-capture system. The latter allows comparing the pose
accuracy of ego-motion estimation algorithms quantitatively. All the data are
released both as standard text files and binary files (i.e., rosbag). This
paper provides an overview of the available data and describes a simulator that
we release open-source to create synthetic event-camera data.

介绍了 Dynamic and Active-pixel Vision sensor（简称 DAVIS）的特点及其在机器人技术和计算机视觉方面的潜在应用，同时公开了一组 DAVIS 数据集，旨在激发研究人员对于高速和高动态范围机器人技术和计算机视觉应用新算法的研究。