Research in unpaired video translation has mainly focused on short-term
temporal consistency by conditioning on neighboring frames. However for
transfer from simulated to photorealistic sequences, available information on
the underlying geometry offers potential for achieving global consistency
across views. We propose a novel approach which combines unpaired image
translation with neural rendering to transfer simulated to photorealistic
surgical abdominal scenes. By introducing global learnable textures and a
lighting-invariant view-consistency loss, our method produces consistent
translations of arbitrary views and thus enables long-term consistent video
synthesis. We design and test our model to generate video sequences from
minimally-invasive surgical abdominal scenes. Because labeled data is often
limited in this domain, photorealistic data where ground truth information from
the simulated domain is preserved is especially relevant. By extending existing
image-based methods to view-consistent videos, we aim to impact the
applicability of simulated training and evaluation environments for surgical
applications. Code and data: this http URL

该文提出了一种结合无配对图像转换和神经渲染的新方法，将模拟的外科腹部场景转化为真实的图像，并实现了长期的一致视频合成。