Conditional image-to-video (cI2V) generation aims to synthesize a new
plausible video starting from an image (e.g., a person's face) and a condition
(e.g., an action class label like smile). The key challenge of the cI2V task
lies in the simultaneous generation of realistic spatial appearance and
temporal dynamics corresponding to the given image and condition. In this
paper, we propose an approach for cI2V using novel latent flow diffusion models
(LFDM) that synthesize an optical flow sequence in the latent space based on
the given condition to warp the given image. Compared to previous
direct-synthesis-based works, our proposed LFDM can better synthesize spatial
details and temporal motion by fully utilizing the spatial content of the given
image and warping it in the latent space according to the generated
temporally-coherent flow. The training of LFDM consists of two separate stages:
(1) an unsupervised learning stage to train a latent flow auto-encoder for
spatial content generation, including a flow predictor to estimate latent flow
between pairs of video frames, and (2) a conditional learning stage to train a
3D-UNet-based diffusion model (DM) for temporal latent flow generation. Unlike
previous DMs operating in pixel space or latent feature space that couples
spatial and temporal information, the DM in our LFDM only needs to learn a
low-dimensional latent flow space for motion generation, thus being more
computationally efficient. We conduct comprehensive experiments on multiple
datasets, where LFDM consistently outperforms prior arts. Furthermore, we show
that LFDM can be easily adapted to new domains by simply finetuning the image
decoder. Our code is available at this https URL

本文提出了一种使用基于新颖潜在流扩散模型（LFDM）的方法，旨在从给定的图像和条件（例如笑容）中合成一个新的合理视频，该模型可以更好地综合空间细节和时间动作。

具有潜在流动扩散模型的条件图像到视频生成

Conditional Image-to-Video Generation with Latent Flow Diffusion Models

Rolling shutter (RS) distortion can be interpreted as the result of picking a
row of pixels from instant global shutter (GS) frames over time during the
exposure of the RS camera. This means that the information of each instant GS
frame is partially, yet sequentially, embedded into the row-dependent
distortion. Inspired by this fact, we address the challenging task of reversing
this process, i.e., extracting undistorted GS frames from images suffering from
RS distortion. However, since RS distortion is coupled with other factors such
as readout settings and the relative velocity of scene elements to the camera,
models that only exploit the geometric correlation between temporally adjacent
images suffer from poor generality in processing data with different readout
settings and dynamic scenes with both camera motion and object motion. In this
paper, instead of two consecutive frames, we propose to exploit a pair of
images captured by dual RS cameras with reversed RS directions for this highly
challenging task. Grounded on the symmetric and complementary nature of dual
reversed distortion, we develop a novel end-to-end model, IFED, to generate
dual optical flow sequence through iterative learning of the velocity field
during the RS time. Extensive experimental results demonstrate that IFED is
superior to naive cascade schemes, as well as the state-of-the-art which
utilizes adjacent RS images. Most importantly, although it is trained on a
synthetic dataset, IFED is shown to be effective at retrieving GS frame
sequences from real-world RS distorted images of dynamic scenes. Code is
available at this https URL

本文提出了一种基于双重滚动快门相机捕捉的成像方法，通过迭代学习速度场来生成双重光流序列，以此提取全局快门帧序列。IFED 模型被证明在处理不同读取设置和带有相机运动和物体运动的动态场景数据方面具有比当前最先进方法更好的表现。