We propose Text2Scene, a method to automatically create realistic textures
for virtual scenes composed of multiple objects. Guided by a reference image
and text descriptions, our pipeline adds detailed texture on labeled 3D
geometries in the room such that the generated colors respect the hierarchical
structure or semantic parts that are often composed of similar materials.
Instead of applying flat stylization on the entire scene at a single step, we
obtain weak semantic cues from geometric segmentation, which are further
clarified by assigning initial colors to segmented parts. Then we add texture
details for individual objects such that their projections on image space
exhibit feature embedding aligned with the embedding of the input. The
decomposition makes the entire pipeline tractable to a moderate amount of
computation resources and memory. As our framework utilizes the existing
resources of image and text embedding, it does not require dedicated datasets
with high-quality textures designed by skillful artists. To the best of our
knowledge, it is the first practical and scalable approach that can create
detailed and realistic textures of the desired style that maintain structural
context for scenes with multiple objects.

通过引用图像和文本描述，我们提出了一种名为 Text2Scene 的方法，可以自动创建由多个物体组成的虚拟场景的逼真纹理，这种方法在房间的标记三维几何体上增加了详细纹理，使生成的颜色尊重经常由类似材料组成的分层结构或语义部分。

Text2Scene：基于文本的室内场景风格化与局部细节

Text2Scene: Text-driven Indoor Scene Stylization with Part-aware Details

In this paper, we propose Text2Scene, a model that generates various forms of
compositional scene representations from natural language descriptions. Unlike
recent works, our method does NOT use Generative Adversarial Networks (GANs).
Text2Scene instead learns to sequentially generate objects and their attributes
(location, size, appearance, etc) at every time step by attending to different
parts of the input text and the current status of the generated scene. We show
that under minor modifications, the proposed framework can handle the
generation of different forms of scene representations, including cartoon-like
scenes, object layouts corresponding to real images, and synthetic images. Our
method is not only competitive when compared with state-of-the-art GAN-based
methods using automatic metrics and superior based on human judgments but also
has the advantage of producing interpretable results.

本文提出了 Text2Scene 模型，该模型从自然语言描述中生成各种形式的组成场景表示。与最近的其他工作不同，Text2Scene 不使用生成对抗网络（GANs），而是通过依次生成对象和它们的属性来学习生成场景，从而实现生成各种形式的场景表示，包括卡通样式的场景、与真实图像相对应的对象布局和合成图像。该方法不仅能够胜任与基于 GAN 的最先进方法相竞争的工作，而且其结果可以被解释。