In this work, we first propose a fully differentiable Many-to-Many (M2M)
splatting framework to interpolate frames efficiently. Given a frame pair, we
estimate multiple bidirectional flows to directly forward warp the pixels to
the desired time step before fusing overlapping pixels. In doing so, each
source pixel renders multiple target pixels and each target pixel can be
synthesized from a larger area of visual context, establishing a many-to-many
splatting scheme with robustness to undesirable artifacts. For each input frame
pair, M2M has a minuscule computational overhead when interpolating an
arbitrary number of in-between frames, hence achieving fast multi-frame
interpolation. However, directly warping and fusing pixels in the intensity
domain is sensitive to the quality of motion estimation and may suffer from
less effective representation capacity. To improve interpolation accuracy, we
further extend an M2M++ framework by introducing a flexible Spatial Selective
Refinement (SSR) component, which allows for trading computational efficiency
for interpolation quality and vice versa. Instead of refining the entire
interpolated frame, SSR only processes difficult regions selected under the
guidance of an estimated error map, thereby avoiding redundant computation.
Evaluation on multiple benchmark datasets shows that our method is able to
improve the efficiency while maintaining competitive video interpolation
quality, and it can be adjusted to use more or less compute as needed.

我们提出了一个全可微的 Many-to-Many（M2M）splatting 框架，通过估计多个双向流来直接向前扭曲像素到所需的时间步骤，实现了多对多的覆盖像素的效果，然后引入了一个灵活的空间选择性优化（SSR）模块来提高插值精度，从而提高了视频插值的效率和质量。

多对多光滑投影和空间选择性细化的视频帧插值

Video Frame Interpolation with Many-to-many Splatting and Spatial  Selective Refinement

Motion-based video frame interpolation commonly relies on optical flow to
warp pixels from the inputs to the desired interpolation instant. Yet due to
the inherent challenges of motion estimation (e.g. occlusions and
discontinuities), most state-of-the-art interpolation approaches require
subsequent refinement of the warped result to generate satisfying outputs,
which drastically decreases the efficiency for multi-frame interpolation. In
this work, we propose a fully differentiable Many-to-Many (M2M) splatting
framework to interpolate frames efficiently. Specifically, given a frame pair,
we estimate multiple bidirectional flows to directly forward warp the pixels to
the desired time step, and then fuse any overlapping pixels. In doing so, each
source pixel renders multiple target pixels and each target pixel can be
synthesized from a larger area of visual context. This establishes a
many-to-many splatting scheme with robustness to artifacts like holes.
Moreover, for each input frame pair, M2M only performs motion estimation once
and has a minuscule computational overhead when interpolating an arbitrary
number of in-between frames, hence achieving fast multi-frame interpolation. We
conducted extensive experiments to analyze M2M, and found that it significantly
improves efficiency while maintaining high effectiveness.

本文提出了一种完全可微的 Many-to-Many（M2M）喷涂框架来高效地进行视频帧插值，该框架对于克服 motion based 视频帧插值的诸如遮挡和不连续性等挑战具有良好的稳健性和高效性，其每个源像素可以呈现多个目标像素，每个目标像素可以从更大的视觉上下文区域综合生成。