As online shopping is growing, the ability for buyers to virtually visualize
products in their settings-a phenomenon we define as "Virtual Try-All"-has
become crucial. Recent diffusion models inherently contain a world model,
rendering them suitable for this task within an inpainting context. However,
traditional image-conditioned diffusion models often fail to capture the
fine-grained details of products. In contrast, personalization-driven models
such as DreamPaint are good at preserving the item's details but they are not
optimized for real-time applications. We present "Diffuse to Choose," a novel
diffusion-based image-conditioned inpainting model that efficiently balances
fast inference with the retention of high-fidelity details in a given reference
item while ensuring accurate semantic manipulations in the given scene content.
Our approach is based on incorporating fine-grained features from the reference
image directly into the latent feature maps of the main diffusion model,
alongside with a perceptual loss to further preserve the reference item's
details. We conduct extensive testing on both in-house and publicly available
datasets, and show that Diffuse to Choose is superior to existing zero-shot
diffusion inpainting methods as well as few-shot diffusion personalization
algorithms like DreamPaint.

基于扩散的图像修复模型实现在线购物时的虚拟可视化，包括细节处理和语义操作。

扩散去选择：在虚拟全尝试中丰富基于图像条件的潜在扩散模型的修复

Diffuse to Choose: Enriching Image Conditioned Inpainting in Latent  Diffusion Models for Virtual Try-All

Recent text-to-image models have achieved impressive results. However, since
they require large-scale datasets of text-image pairs, it is impractical to
train them on new domains where data is scarce or not labeled. In this work, we
propose using large-scale retrieval methods, in particular, efficient
k-Nearest-Neighbors (kNN), which offers novel capabilities: (1) training a
substantially small and efficient text-to-image diffusion model without any
text, (2) generating out-of-distribution images by simply swapping the
retrieval database at inference time, and (3) performing text-driven local
semantic manipulations while preserving object identity. To demonstrate the
robustness of our method, we apply our kNN approach on two state-of-the-art
diffusion backbones, and show results on several different datasets. As
evaluated by human studies and automatic metrics, our method achieves
state-of-the-art results compared to existing approaches that train
text-to-image generation models using images only (without paired text data)

通过使用大规模检索方法（尤其是高效的 k - 最近邻），我们提出了一种小型高效的文本到图像扩散模型，该模型不需要文本，并且可以在保持对象身份的同时执行基于文本驱动的局部语义操作。我们的方法在多个数据集上实现了最先进的结果，并且与使用仅图像而没有配对文本数据进行文本到图像生成模型训练的现有方法相比，在人类研究和自动指标方面都取得了最先进的结果。