In recent years, advances in text-to-image (T2I) diffusion models have
substantially elevated the quality of their generated images. However,
achieving fine-grained control over attributes remains a challenge due to the
limitations of natural language prompts (such as no continuous set of
intermediate descriptions existing between ``person'' and ``old person''). Even
though many methods were introduced that augment the model or generation
process to enable such control, methods that do not require a fixed reference
image are limited to either enabling global fine-grained attribute expression
control or coarse attribute expression control localized to specific subjects,
not both simultaneously. We show that there exist directions in the commonly
used token-level CLIP text embeddings that enable fine-grained subject-specific
control of high-level attributes in text-to-image models. Based on this
observation, we introduce one efficient optimization-free and one robust
optimization-based method to identify these directions for specific attributes
from contrastive text prompts. We demonstrate that these directions can be used
to augment the prompt text input with fine-grained control over attributes of
specific subjects in a compositional manner (control over multiple attributes
of a single subject) without having to adapt the diffusion model. Project page:
this https URL Code is available at
this https URL

通过观察在常用的基于标记级别的 CLIP 文本嵌入中存在与高级属性进行主题特定的精细控制的方向，我们提出了一种有效的无优化方法和一种稳健的优化方法，以从对比文本提示中识别出特定属性的这些方向。我们演示了这些方向可以用于以组合方式增强提示文本输入，实现对特定主题的属性进行精细控制（单个主题的多个属性控制），而无需适应扩散模型。

通过识别语义方向在 T2I 模型中实现连续、个体化的属性控制

Continuous, Subject-Specific Attribute Control in T2I Models by  Identifying Semantic Directions

Reconstructing visual stimuli from measured functional magnetic resonance
imaging (fMRI) has been a meaningful and challenging task. Previous studies
have successfully achieved reconstructions with structures similar to the
original images, such as the outlines and size of some natural images. However,
these reconstructions lack explicit semantic information and are difficult to
discern. In recent years, many studies have utilized multi-modal pre-trained
models with stronger generative capabilities to reconstruct images that are
semantically similar to the original ones. However, these images have
uncontrollable structural information such as position and orientation. To
address both of the aforementioned issues simultaneously, we propose a
two-stage image reconstruction model called MindDiffuser, utilizing Stable
Diffusion. In Stage 1, the VQ-VAE latent representations and the CLIP text
embeddings decoded from fMRI are put into the image-to-image process of Stable
Diffusion, which yields a preliminary image that contains semantic and
structural information. In Stage 2, we utilize the low-level CLIP visual
features decoded from fMRI as supervisory information, and continually adjust
the two features in Stage 1 through backpropagation to align the structural
information. The results of both qualitative and quantitative analyses
demonstrate that our proposed model has surpassed the current state-of-the-art
models in terms of reconstruction results on Natural Scenes Dataset (NSD).
Furthermore, the results of ablation experiments indicate that each component
of our model is effective for image reconstruction.

本文提出了一种名为 MindDiffuser 的两阶段图像重构模型，利用了稳定扩散技术，成功地同时实现了语义和结构信息的重构，并在自然场景数据集方面的重建结果超过了当前最先进模型。