Videos obtained by rolling-shutter (RS) cameras result in spatially-distorted
frames. These distortions become significant under fast camera/scene motions.
Undoing effects of RS is sometimes addressed as a spatial problem, where
objects need to be rectified/displaced in order to generate their correct
global shutter (GS) frame. However, the cause of the RS effect is inherently
temporal, not spatial. In this paper we propose a space-time solution to the RS
problem. We observe that despite the severe differences between their xy
frames, a RS video and its corresponding GS video tend to share the exact same
xt slices -- up to a known sub-frame temporal shift. Moreover, they share the
same distribution of small 2D xt-patches, despite the strong temporal aliasing
within each video. This allows to constrain the GS output video using
video-specific constraints imposed by the RS input video. Our algorithm is
composed of 3 main components: (i) Dense temporal upsampling between
consecutive RS frames using an off-the-shelf method, (which was trained on
regular video sequences), from which we extract GS "proposals". (ii) Learning
to correctly merge an ensemble of such GS "proposals" using a dedicated
MergeNet. (iii) A video-specific zero-shot optimization which imposes the
similarity of xt-patches between the GS output video and the RS input video.
Our method obtains state-of-the-art results on benchmark datasets, both
numerically and visually, despite being trained on a small synthetic RS/GS
dataset. Moreover, it generalizes well to new complex RS videos with motion
types outside the distribution of the training set (e.g., complex non-rigid
motions) -- videos which competing methods trained on much more data cannot
handle well. We attribute these generalization capabilities to the combination
of external and internal constraints.

本研究提出了一种基于空时解决方案和视频特定优化的方法，通过稠密的时间上采样获得 GS “提议”，使用 MergeNet 将多个 GS “提议” 合并成为 GS 输出视频，并通过零样调整使输出视频与输入视频的 xt 块相似，从而解决了滚动快门相机的视频图像失真问题。

结合内部和外部限制展开视频快门

Combining Internal and External Constraints for Unrolling Shutter in  Videos

Abrupt motion of camera or objects in a scene result in a blurry video, and
therefore recovering high quality video requires two types of enhancements:
visual enhancement and temporal upsampling. A broad range of research attempted
to recover clean frames from blurred image sequences or temporally upsample
frames by interpolation, yet there are very limited studies handling both
problems jointly. In this work, we present a novel framework for deblurring,
interpolating and extrapolating sharp frames from a motion-blurred video in an
end-to-end manner. We design our framework by first learning the pixel-level
motion that caused the blur from the given inputs via optical flow estimation
and then predict multiple clean frames by warping the decoded features with the
estimated flows. To ensure temporal coherence across predicted frames and
address potential temporal ambiguity, we propose a simple, yet effective
flow-based rule. The effectiveness and favorability of our approach are
highlighted through extensive qualitative and quantitative evaluations on
motion-blurred datasets from high speed videos.

本研究提出了一种新的框架，该框架可以通过光流估计学习引起模糊的像素级运动，然后将已解码的特征变形以估算的光流预测多个清晰的帧，从而以一种端到端的方式从运动模糊视频中去模糊、插值和外推锐化帧。