Visual layout plays a critical role in graphic design fields such as
advertising, posters, and web UI design. The recent trend towards content-aware
layout generation through generative models has shown promise, yet it often
overlooks the semantic intricacies of layout design by treating it as a simple
numerical optimization. To bridge this gap, we introduce PosterLlama, a network
designed for generating visually and textually coherent layouts by reformatting
layout elements into HTML code and leveraging the rich design knowledge
embedded within language models. Furthermore, we enhance the robustness of our
model with a unique depth-based poster augmentation strategy. This ensures our
generated layouts remain semantically rich but also visually appealing, even
with limited data. Our extensive evaluations across several benchmarks
demonstrate that PosterLlama outperforms existing methods in producing
authentic and content-aware layouts. It supports an unparalleled range of
conditions, including but not limited to unconditional layout generation,
element conditional layout generation, layout completion, among others, serving
as a highly versatile user manipulation tool.

PosterLlama 是一种生成视觉上连贯且文本相关的布局的网络，通过重新格式化布局元素为 HTML 代码并利用语言模型中嵌入的丰富设计知识来进行设计，通过独特的基于深度的海报增强策略提高模型的鲁棒性。在多个基准测试中，PosterLlama 在生成真实且内容感知的布局方面优于现有方法，支持无条件布局生成、元素条件布局生成、布局完成等条件，并作为一种高度灵活的用户操作工具。

PosterLlama：连接语言模型的设计能力与内容感知布局生成

PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware  Layout Generation

Recent studies on machine reading comprehension have focused on text-level
understanding but have not yet reached the level of human understanding of the
visual layout and content of real-world documents. In this study, we introduce
a new visual machine reading comprehension dataset, named VisualMRC, wherein
given a question and a document image, a machine reads and comprehends texts in
the image to answer the question in natural language. Compared with existing
visual question answering (VQA) datasets that contain texts in images,
VisualMRC focuses more on developing natural language understanding and
generation abilities. It contains 30,000+ pairs of a question and an
abstractive answer for 10,000+ document images sourced from multiple domains of
webpages. We also introduce a new model that extends existing
sequence-to-sequence models, pre-trained with large-scale text corpora, to take
into account the visual layout and content of documents. Experiments with
VisualMRC show that this model outperformed the base sequence-to-sequence
models and a state-of-the-art VQA model. However, its performance is still
below that of humans on most automatic evaluation metrics. The dataset will
facilitate research aimed at connecting vision and language understanding.

本研究提出了一个名为 VisualMRC 的新的视觉机器阅读理解数据集，包含来自多个网络域的 10,000 多个文档图像，以及 30,000 多个问题和答案对，以帮助发展自然语言理解和生成能力。此外，研究还引入了一个扩展了现有序列到序列模型的新模型，以考虑文档的视觉布局和内容。该模型在自动评估度量方面优于基本序列到序列模型和最先进的 VQA 模型，但其性能仍低于大多数人类表现。该数据集将有助于将视觉和语言理解联系起来的研究。