Different conditional video prediction tasks, like video future frame
prediction and video frame interpolation, are normally solved by task-related
models even though they share many common underlying characteristics.
Furthermore, almost all conditional video prediction models can only achieve
discrete prediction. In this paper, we propose a unified model that addresses
these two issues at the same time. We show that conditional video prediction
can be formulated as a neural process, which maps input spatio-temporal
coordinates to target pixel values given context spatio-temporal coordinates
and context pixel values. Specifically, we feed the implicit neural
representation of coordinates and context pixel features into a
Transformer-based non-autoregressive conditional video prediction model. Our
task-specific models outperform previous work for video future frame prediction
and video interpolation on multiple datasets. Importantly, the model is able to
interpolate or predict with an arbitrary high frame rate, i.e., continuous
prediction. Our source code is available at https://npvp.github.io.

本文提出了一种统一的神经过程模型，该模型以强化了的 Transformer 为基础，既可以在多个数据集上有效地处理条件视频预测任务（包括视频未来帧预测和视频帧插值），又可以实现连续预测 / 插值，且性能优于现有工作。