Video frame interpolation aims to synthesize one or multiple frames between
two consecutive frames in a video. It has a wide range of applications
including slow-motion video generation, frame-rate up-scaling and developing
video codecs. Some older works tackled this problem by assuming per-pixel
linear motion between video frames. However, objects often follow a non-linear
motion pattern in the real domain and some recent methods attempt to model
per-pixel motion by non-linear models (e.g., quadratic). A quadratic model can
also be inaccurate, especially in the case of motion discontinuities over time
(i.e. sudden jerks) and occlusions, where some of the flow information may be
invalid or inaccurate.
In our paper, we propose to approximate the per-pixel motion using a
space-time convolution network that is able to adaptively select the motion
model to be used. Specifically, we are able to softly switch between a linear
and a quadratic model. Towards this end, we use an end-to-end 3D CNN
encoder-decoder architecture over bidirectional optical flows and occlusion
maps to estimate the non-linear motion model of each pixel. Further, a motion
refinement module is employed to refine the non-linear motion and the
interpolated frames are estimated by a simple warping of the neighboring frames
with the estimated per-pixel motion. Through a set of comprehensive
experiments, we validate the effectiveness of our model and show that our
method outperforms state-of-the-art algorithms on four datasets (Vimeo, DAVIS,
HD and GoPro).

本文提出了一种适应性选择运动模型的时空卷积网络来逼近像素间的运动。通过实验结果证实算法在四个数据集上的表现均优于现有算法，尤其是在处理非线性运动和运动不连续的情况下具有显著优势。