Colorectal cancer screening modalities, such as optical colonoscopy (OC) and
virtual colonoscopy (VC), are critical for diagnosing and ultimately removing
polyps (precursors of colon cancer). The non-invasive VC is normally used to
inspect a 3D reconstructed colon (from CT scans) for polyps and if found, the
OC procedure is performed to physically traverse the colon via endoscope and
remove these polyps. In this paper, we present a deep learning framework,
Extended and Directional CycleGAN, for lossy unpaired image-to-image
translation between OC and VC to augment OC video sequences with
scale-consistent depth information from VC, and augment VC with
patient-specific textures, color and specular highlights from OC (e.g, for
realistic polyp synthesis). Both OC and VC contain structural information, but
it is obscured in OC by additional patient-specific texture and specular
highlights, hence making the translation from OC to VC lossy. The existing
CycleGAN approaches do not handle lossy transformations. To address this
shortcoming, we introduce an extended cycle consistency loss, which compares
the geometric structures from OC in the VC domain. This loss removes the need
for the CycleGAN to embed OC information in the VC domain. To handle a stronger
removal of the textures and lighting, a Directional Discriminator is introduced
to differentiate the direction of translation (by creating paired information
for the discriminator), as opposed to the standard CycleGAN which is
direction-agnostic. Combining the extended cycle consistency loss and the
Directional Discriminator, we show state-of-the-art results on scale-consistent
depth inference for phantom, textured VC and for real polyp and normal colon
video sequences. We also present results for realistic pendunculated and flat
polyp synthesis from bumps introduced in 3D VC models. Code/models:
this https URL

使用深度学习框架，本研究提出了一种新的方法来实现 OC 和 VC 之间的图像 - 图像转换，并演示了在虚拟结肠镜检查，真实息肉和正常结肠视频序列以及 3D VC 模型中创造逼真的息肉的结果。