The diffusion model has been proven a powerful generative model in recent
years, yet remains a challenge in generating visual text. Several methods
alleviated this issue by incorporating explicit text position and content as
guidance on where and what text to render. However, these methods still suffer
from several drawbacks, such as limited flexibility and automation, constrained
capability of layout prediction, and restricted style diversity. In this paper,
we present TextDiffuser-2, aiming to unleash the power of language models for
text rendering. Firstly, we fine-tune a large language model for layout
planning. The large language model is capable of automatically generating
keywords for text rendering and also supports layout modification through
chatting. Secondly, we utilize the language model within the diffusion model to
encode the position and texts at the line level. Unlike previous methods that
employed tight character-level guidance, this approach generates more diverse
text images. We conduct extensive experiments and incorporate user studies
involving human participants as well as GPT-4V, validating TextDiffuser-2's
capacity to achieve a more rational text layout and generation with enhanced
diversity. The code and model will be available at
https://aka.ms/textdiffuser-2.

通过使用语言模型进行布局规划，TextDiffuser-2 能够实现更合理的文本布局和具有增强多样性的生成。

TextDiffuser-2：释放语言模型在文本呈现中的能力

TextDiffuser-2: Unleashing the Power of Language Models for Text  Rendering

A 3D scene consists of a set of objects, each with a shape and a layout
giving their position in space. Understanding 3D scenes from 2D images is an
important goal, with applications in robotics and graphics. While there have
been recent advances in predicting 3D shape and layout from a single image,
most approaches rely on 3D ground truth for training which is expensive to
collect at scale. We overcome these limitations and propose a method that
learns to predict 3D shape and layout for objects without any ground truth
shape or layout information: instead we rely on multi-view images with 2D
supervision which can more easily be collected at scale. Through extensive
experiments on 3D Warehouse, Hypersim, and ScanNet we demonstrate that our
approach scales to large datasets of realistic images, and compares favorably
to methods relying on 3D ground truth. On Hypersim and ScanNet where reliable
3D ground truth is not available, our approach outperforms supervised
approaches trained on smaller and less diverse datasets.

本研究提出了一种基于多视图图像的方法，用于识别三维场景的对象形状和布局，通过在大型数据集上的实验，证明该方法可扩展应用于现实图像，并与依赖三维信息的方法相比，表现得更好。

学习三维物体的形状和布局，无需三维监督

Learning 3D Object Shape and Layout without 3D Supervision

Creating presentation materials requires complex multimodal reasoning skills
to summarize key concepts and arrange them in a logical and visually pleasing
manner. Can machines learn to emulate this laborious process? We present a
novel task and approach for document-to-slide generation. Solving this involves
document summarization, image and text retrieval, slide structure and layout
prediction to arrange key elements in a form suitable for presentation. We
propose a hierarchical sequence-to-sequence approach to tackle our task in an
end-to-end manner. Our approach exploits the inherent structures within
documents and slides and incorporates paraphrasing and layout prediction
modules to generate slides. To help accelerate research in this domain, we
release a dataset about 6K paired documents and slide decks used in our
experiments. We show that our approach outperforms strong baselines and
produces slides with rich content and aligned imagery.

文档导出幻灯片是一项困难的多模态推理任务，我们提出了一种新颖的基于层次序列 - 序列算法的方法，在文档摘要、图像和文本检索、幻灯片结构和布局预测等方面具有较高的性能。