Style transfer is a promising approach to close the sim-to-real gap in
medical endoscopy. Rendering realistic endoscopic videos by traversing
pre-operative scans (such as MRI or CT) can generate realistic simulations as
well as ground truth camera poses and depth maps. Although image-to-image (I2I)
translation models such as CycleGAN perform well, they are unsuitable for
video-to-video synthesis due to the lack of temporal consistency, resulting in
artifacts between frames. We propose MeshBrush, a neural mesh stylization
method to synthesize temporally consistent videos with differentiable
rendering. MeshBrush uses the underlying geometry of patient imaging data while
leveraging existing I2I methods. With learned per-vertex textures, the stylized
mesh guarantees consistency while producing high-fidelity outputs. We
demonstrate that mesh stylization is a promising approach for creating
realistic simulations for downstream tasks such as training and preoperative
planning. Although our method is tested and designed for ureteroscopy, its
components are transferable to general endoscopic and laparoscopic procedures.

通过神经网格风格化的方法，使用病人成像数据的基础几何特征以及现有的图像转换方法，实现产生高保真、具有时间一致性的视频，可用于训练和术前规划等下游任务。

MeshBrush: 内窥镜的神经风格化解剖网格绘制

MeshBrush: Painting the Anatomical Mesh with Neural Stylization for  Endoscopy

We propose CLIP-Actor, a text-driven motion recommendation and neural mesh
stylization system for human mesh animation. CLIP-Actor animates a 3D human
mesh to conform to a text prompt by recommending a motion sequence and
optimizing mesh style attributes. We build a text-driven human motion
recommendation system by leveraging a large-scale human motion dataset with
language labels. Given a natural language prompt, CLIP-Actor suggests a
text-conforming human motion in a coarse-to-fine manner. Then, our novel
zero-shot neural style optimization detailizes and texturizes the recommended
mesh sequence to conform to the prompt in a temporally-consistent and
pose-agnostic manner. This is distinctive in that prior work fails to generate
plausible results when the pose of an artist-designed mesh does not conform to
the text from the beginning. We further propose the spatio-temporal view
augmentation and mask-weighted embedding attention, which stabilize the
optimization process by leveraging multi-frame human motion and rejecting
poorly rendered views. We demonstrate that CLIP-Actor produces plausible and
human-recognizable style 3D human mesh in motion with detailed geometry and
texture solely from a natural language prompt.

提出了 CLIP-Actor：一种基于文本的动作推荐和神经网格风格化系统，能够根据文本提示为 3D 人类网格动画建议运动序列和优化网格样式属性，这对于以前无法生成符合自然语言提示的 3D 人物运动效果是非常独特的，CLIP-Actor 可以仅仅通过自然语言提示产生具有细节几何和纹理的真人可识别的风格 3D 人类网格运动。