The vanilla image completion approaches are sensitive to the large missing
regions due to limited available reference information for plausible
generation. To mitigate this, existing methods incorporate the extra cue as a
guidance for image completion. Despite improvements, these approaches are often
restricted to employing a single modality (e.g., segmentation or sketch maps),
which lacks scalability in leveraging multi-modality for more plausible
completion. In this paper, we propose a novel, simple yet effective method for
Multi-modal Guided Image Completion, dubbed MaGIC, which not only supports a
wide range of single modality as the guidance (e.g., text, canny edge, sketch,
segmentation, reference image, depth, and pose), but also adapts to arbitrarily
customized combination of these modalities (i.e., arbitrary multi-modality) for
image completion. For building MaGIC, we first introduce a modality-specific
conditional U-Net (MCU-Net) that injects single-modal signal into a U-Net
denoiser for single-modal guided image completion. Then, we devise a consistent
modality blending (CMB) method to leverage modality signals encoded in multiple
learned MCU-Nets through gradient guidance in latent space. Our CMB is
training-free, and hence avoids the cumbersome joint re-training of different
modalities, which is the secret of MaGIC to achieve exceptional flexibility in
accommodating new modalities for completion. Experiments show the superiority
of MaGIC over state-of-arts and its generalization to various completion tasks
including in/out-painting and local editing. Our project with code and models
is available at yeates.github.io/MaGIC-Page/.

本文提出了一种名为 MaGIC 的多模态引导图像补全方法，该方法不仅支持单模态引导，还能适应任意多种模态的引导，使用模态特定的 MCU-Net 注入单一信号，再通过一致性模态混合 (CMB) 方法传播梯度，实现了在没有大量参考信息的情况下的高质量图像补全。

MaGIC: 多模态导向图像修复

MaGIC: Multi-modality Guided Image Completion

Incorporating a human-in-the-loop system when deploying automated decision
support is critical in healthcare contexts to create trust, as well as provide
reliable performance on a patient-to-patient basis. Deep learning methods while
having high performance, do not allow for this patient-centered approach due to
the lack of uncertainty representation. Thus, we present a framework of
uncertainty representation evaluated for medical image segmentation, using
MCU-Net which combines a U-Net with Monte Carlo Dropout, evaluated with four
different uncertainty metrics. The framework augments this by adding a
human-in-the-loop aspect based on an uncertainty threshold for automated
referral of uncertain cases to a medical professional. We demonstrate that
MCU-Net combined with epistemic uncertainty and an uncertainty threshold tuned
for this application maximizes automated performance on an individual patient
level, yet refers truly uncertain cases. This is a step towards uncertainty
representations when deploying machine learning based decision support in
healthcare settings.

该研究通过在自动化决策支持系统中引入人为因素，建立了一种以病人为中心的不确定性表示框架，并使用 MCU-Net 和四种不同的不确定性度量对其进行了评估，为医学图像分割提供了更可靠的性能。同样，该框架基于不确定性阈值添加了人在循环，使得机器学习模型达到最大的个体化表现，同时将真实的不确定案例传递给专业医生，进而在医疗领域机器学习决策支持中迈出了一步。