Recent advancements in instruction-following models have made user interactions with models more user-friendly and efficient, broadening their applicability. In graphic design, non-professional users often struggle to create visually appealing layouts due to limited skills and resources. In this work, we introduce a novel multimodal instruction-following framework for layout planning, allowing users to easily arrange visual elements into tailored layouts by specifying canvas size and design purpose, such as for book covers, posters, brochures, or menus. We developed three layout reasoning tasks to train the model in understanding and executing layout instructions. Experiments on two benchmarks show that our method not only simplifies the design process for non-professionals but also surpasses the performance of few-shot GPT-4V models, with mIoU higher by 12% on Crello. This progress highlights the potential of multimodal instruction-following models to automate and simplify the design process, providing an approachable solution for a wide range of design tasks on visually-rich documents.

近期，在指令追踪模型方面取得的进展使得用户与模型的交互更加用户友好和高效，扩大了它们的适用范围。本研究引入了一种新颖的多模态指令追踪框架，用于布局规划，在设计领域，非专业用户经常由于技能和资源有限而难以创建具有视觉吸引力的布局。我们提出了三个布局推理任务来训练模型理解和执行布局指令。在两个基准测试中的实验证明，我们的方法不仅简化了非专业人员的设计过程，而且在Crello上的mIoU高于few-shot GPT-4V模型12%。这一进展凸显了多模态指令追踪模型在自动化和简化设计过程方面的潜力，为视觉丰富文档上的各类设计任务提供了可行的解决方案。

视觉丰富文档的自动布局规划与指导模型