Using image as prompts for 3D generation demonstrate particularly strong
performances compared to using text prompts alone, for images provide a more
intuitive guidance for the 3D generation process. In this work, we delve into
the potential of using multiple image prompts, instead of a single image
prompt, for 3D generation. Specifically, we build on ImageDream, a novel
image-prompt multi-view diffusion model, to support multi-view images as the
input prompt. Our method, dubbed MultiImageDream, reveals that transitioning
from a single-image prompt to multiple-image prompts enhances the performance
of multi-view and 3D object generation according to various quantitative
evaluation metrics and qualitative assessments. This advancement is achieved
without the necessity of fine-tuning the pre-trained ImageDream multi-view
diffusion model.

使用多个图像提示而非单一图像提示可以提高多视图和 3D 物体生成的性能，无需对预训练的 ImageDream 多视图扩散模型进行微调。

多视图图像引导的多视图扩散用于改善 3D 生成

Multi-view Image Prompted Multi-view Diffusion for Improved 3D  Generation

Large-scale Text-to-Image (T2I) diffusion models have revolutionized image
generation over the last few years. Although owning diverse and high-quality
generation capabilities, translating these abilities to fine-grained image
editing remains challenging. In this paper, we propose DiffEditor to rectify
two weaknesses in existing diffusion-based image editing: (1) in complex
scenarios, editing results often lack editing accuracy and exhibit unexpected
artifacts; (2) lack of flexibility to harmonize editing operations, e.g.,
imagine new content. In our solution, we introduce image prompts in
fine-grained image editing, cooperating with the text prompt to better describe
the editing content. To increase the flexibility while maintaining content
consistency, we locally combine stochastic differential equation (SDE) into the
ordinary differential equation (ODE) sampling. In addition, we incorporate
regional score-based gradient guidance and a time travel strategy into the
diffusion sampling, further improving the editing quality. Extensive
experiments demonstrate that our method can efficiently achieve
state-of-the-art performance on various fine-grained image editing tasks,
including editing within a single image (e.g., object moving, resizing, and
content dragging) and across images (e.g., appearance replacing and object
pasting). Our source code is released at
this https URL

我们提出了 DiffEditor 方法，利用图像提示和文本提示来改善细粒度的图像编辑，通过在扩散采样中引入局部组合的随机微分方程 (SDE)，梯度引导和时间旅行策略，我们的方法在各种细粒度图像编辑任务中取得了最先进的性能。

DiffEditor：基于扩散的图像编辑中的准确性和灵活性增强

DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image  Editing

Text-driven 3D scene editing has gained significant attention owing to its
convenience and user-friendliness. However, existing methods still lack
accurate control of the specified appearance and location of the editing result
due to the inherent limitations of the text description. To this end, we
propose a 3D scene editing framework, TIPEditor, that accepts both text and
image prompts and a 3D bounding box to specify the editing region. With the
image prompt, users can conveniently specify the detailed appearance/style of
the target content in complement to the text description, enabling accurate
control of the appearance. Specifically, TIP-Editor employs a stepwise 2D
personalization strategy to better learn the representation of the existing
scene and the reference image, in which a localization loss is proposed to
encourage correct object placement as specified by the bounding box.
Additionally, TIPEditor utilizes explicit and flexible 3D Gaussian splatting as
the 3D representation to facilitate local editing while keeping the background
unchanged. Extensive experiments have demonstrated that TIP-Editor conducts
accurate editing following the text and image prompts in the specified bounding
box region, consistently outperforming the baselines in editing quality, and
the alignment to the prompts, qualitatively and quantitatively.

TIPEditor 是一个接受文本和图像提示以及 3D 边界框来规定编辑区域的 3D 场景编辑框架，通过使用图像提示用户可以方便地指定目标内容的详细外观或样式，实现准确控制；TIPEditor 在现有场景和参考图像的表示之间采用逐步的 2D 个性化策略来更好地学习，通过定位损失鼓励正确的对象布局；此外，TIPEditor 利用显式和灵活的 3D 高斯重建作为 3D 表示，以便进行局部编辑而保持背景不变。广泛的实验证明，TIP-Editor 能够按照文本和图像提示在指定的边界框区域进行准确的编辑，无论从编辑质量还是与提示的对齐程度来看，在定性和定量方面都优于基线模型。

TIP-Editor: 一个基于文本和图片提示的准确 3D 编辑器

TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And  Image-Prompts

Text-driven video generation witnesses rapid progress. However, merely using
text prompts is not enough to depict the desired subject appearance that
accurately aligns with users' intents, especially for customized content
creation. In this paper, we study the task of video generation with image
prompts, which provide more accurate and direct content control beyond the text
prompts. Specifically, we propose a feed-forward framework VideoBooth, with two
dedicated designs: 1) We propose to embed image prompts in a coarse-to-fine
manner. Coarse visual embeddings from image encoder provide high-level
encodings of image prompts, while fine visual embeddings from the proposed
attention injection module provide multi-scale and detailed encoding of image
prompts. These two complementary embeddings can faithfully capture the desired
appearance. 2) In the attention injection module at fine level, multi-scale
image prompts are fed into different cross-frame attention layers as additional
keys and values. This extra spatial information refines the details in the
first frame and then it is propagated to the remaining frames, which maintains
temporal consistency. Extensive experiments demonstrate that VideoBooth
achieves state-of-the-art performance in generating customized high-quality
videos with subjects specified in image prompts. Notably, VideoBooth is a
generalizable framework where a single model works for a wide range of image
prompts with feed-forward pass.

本文介绍了一种使用图像提示进行视频生成的方法，通过粗粒度到细粒度的图像嵌入和细粒度的注意力注入模块，实现了对指定主题的高质量视频生成。