Style transfer is a promising approach to close the sim-to-real gap in
medical endoscopy. Rendering realistic endoscopic videos by traversing
pre-operative scans (such as MRI or CT) can generate realistic simulations as
well as ground truth camera poses and depth maps. Although image-to-image (I2I)
translation models such as CycleGAN perform well, they are unsuitable for
video-to-video synthesis due to the lack of temporal consistency, resulting in
artifacts between frames. We propose MeshBrush, a neural mesh stylization
method to synthesize temporally consistent videos with differentiable
rendering. MeshBrush uses the underlying geometry of patient imaging data while
leveraging existing I2I methods. With learned per-vertex textures, the stylized
mesh guarantees consistency while producing high-fidelity outputs. We
demonstrate that mesh stylization is a promising approach for creating
realistic simulations for downstream tasks such as training and preoperative
planning. Although our method is tested and designed for ureteroscopy, its
components are transferable to general endoscopic and laparoscopic procedures.

通过神经网格风格化的方法，使用病人成像数据的基础几何特征以及现有的图像转换方法，实现产生高保真、具有时间一致性的视频，可用于训练和术前规划等下游任务。