The three areas of realistic forward rendering, per-pixel inverse rendering,
and generative image synthesis may seem like separate and unrelated sub-fields
of graphics and vision. However, recent work has demonstrated improved
estimation of per-pixel intrinsic channels (albedo, roughness, metallicity)
based on a diffusion architecture; we call this the RGB$\rightarrow$X problem.
We further show that the reverse problem of synthesizing realistic images given
intrinsic channels, X$\rightarrow$RGB, can also be addressed in a diffusion
framework.
Focusing on the image domain of interior scenes, we introduce an improved
diffusion model for RGB$\rightarrow$X, which also estimates lighting, as well
as the first diffusion X$\rightarrow$RGB model capable of synthesizing
realistic images from (full or partial) intrinsic channels. Our
X$\rightarrow$RGB model explores a middle ground between traditional rendering
and generative models: we can specify only certain appearance properties that
should be followed, and give freedom to the model to hallucinate a plausible
version of the rest.
This flexibility makes it possible to use a mix of heterogeneous training
datasets, which differ in the available channels. We use multiple existing
datasets and extend them with our own synthetic and real data, resulting in a
model capable of extracting scene properties better than previous work and of
generating highly realistic images of interior scenes.

通过扩散模型以及逆渲染和生成图像合成的方法，我们提出了改进的模型来实现逆向渲染问题 RGB→X 和合成问题 X→RGB 的效果。我们的模型在室内场景图像领域具有高度的灵活性和真实性。

RGB↔X：使用材质和光照感知扩散模型的图像分解与合成

RGB$\leftrightarrow$X: Image decomposition and synthesis using material-  and lighting-aware diffusion models

Despite significant progress in generative image synthesis and full-body
generation in particular, state-of-the-art methods are either
context-independent, overly reliant to text prompts, or bound to the curated
training datasets, such as fashion images with monotonous backgrounds. Here,
our goal is to generate people in clothing that is semantically appropriate for
a given scene. To this end, we present ESP, a novel method for context-aware
full-body generation, that enables photo-realistic inpainting of people into
existing "in-the-wild" photographs. ESP is conditioned on a 2D pose and
contextual cues that are extracted from the environment photograph and
integrated into the generation process. Our models are trained on a dataset
containing a set of in-the-wild photographs of people covering a wide range of
different environments. The method is analyzed quantitatively and
qualitatively, and we show that ESP outperforms state-of-the-art on the task of
contextual full-body generation.

通过在给定场景中融入 2D 姿势和上下文提示，提出了一种用于上下文感知全身生成的新方法 ESP，该方法在现有的 “野外” 照片中实现了真实感人物修复，定量和定性分析表明 ESP 在上下文全身生成的任务上优于现有的方法。

特定环境下的人群

Environment-Specific People

Novel architectures have recently improved generative image synthesis leading
to excellent visual quality in various tasks. Of particular note is the field
of ``AI-Art'', which has seen unprecedented growth with the emergence of
powerful multimodal models such as CLIP. By combining speech and image
synthesis models, so-called ``prompt-engineering'' has become established, in
which carefully selected and composed sentences are used to achieve a certain
visual style in the synthesized image. In this note, we present an alternative
approach based on retrieval-augmented diffusion models (RDMs). In RDMs, a set
of nearest neighbors is retrieved from an external database during training for
each training instance, and the diffusion model is conditioned on these
informative samples. During inference (sampling), we replace the retrieval
database with a more specialized database that contains, for example, only
images of a particular visual style. This provides a novel way to prompt a
general trained model after training and thereby specify a particular visual
style. As shown by our experiments, this approach is superior to specifying the
visual style within the text prompt. We open-source code and model weights at
this https URL .

通过检索增强的扩散模型，我们提出了一种替代性方法，改进了生成图像合成技术，优于基于文本提示的视觉样式规定。