We consider the problem of constraining diffusion model outputs with a
user-supplied reference image. Our key objective is to extract multiple
attributes (e.g., color, object, layout, style) from this single reference
image, and then generate new samples with them. One line of existing work
proposes to invert the reference images into a single textual conditioning
vector, enabling generation of new samples with this learned token. These
methods, however, do not learn multiple tokens that are necessary to condition
model outputs on the multiple attributes noted above. Another line of
techniques expand the inversion space to learn multiple embeddings but they do
this only along the layer dimension (e.g., one per layer of the DDPM model) or
the timestep dimension (one for a set of timesteps in the denoising process),
leading to suboptimal attribute disentanglement. To address the aforementioned
gaps, the first contribution of this paper is an extensive analysis to
determine which attributes are captured in which dimension of the denoising
process. As noted above, we consider both the time-step dimension (in reverse
denoising) as well as the DDPM model layer dimension. We observe that often a
subset of these attributes are captured in the same set of model layers and/or
across same denoising timesteps. For instance, color and style are captured
across same U-Net layers, whereas layout and color are captured across same
timestep stages. Consequently, an inversion process that is designed only for
the time-step dimension or the layer dimension is insufficient to disentangle
all attributes. This leads to our second contribution where we design a new
multi-attribute inversion algorithm, MATTE, with associated
disentanglement-enhancing regularization losses, that operates across both
dimensions and explicitly leads to four disentangled tokens (color, style,
layout, and object).

本文研究了使用用户提供的参考图像约束扩散模型输出的问题，旨在从单个参考图像中提取多个属性，并使用这些属性生成新的样本。首先，文中分析了在去噪过程的时间步维度和 DDPM 模型层维度中捕获的属性，并发现某些属性在相同的模型层和时间步骤中被捕获。基于这些发现，文中设计了一种新的多属性逆向算法 MATTE，可以明确分离四个属性（颜色、风格、布局和物体）。