Websites are critical in today's digital world, with over 1.11 billion
currently active and approximately 252,000 new sites launched daily. Converting
website layout design into functional UI code is a time-consuming yet
indispensable step of website development. Manual methods of converting visual
designs into functional code present significant challenges, especially for
non-experts. To explore automatic design-to-code solutions, we first conduct a
motivating study on GPT-4o and identify three types of issues in generating UI
code: element omission, element distortion, and element misarrangement. We
further reveal that a focus on smaller visual segments can help multimodal
large language models (MLLMs) mitigate these failures in the generation
process. In this paper, we propose DCGen, a divide-and-conquer-based approach
to automate the translation of webpage design to UI code. DCGen starts by
dividing screenshots into manageable segments, generating descriptions for each
segment, and then reassembling them into complete UI code for the entire
screenshot. We conduct extensive testing with a dataset comprised of real-world
websites and various MLLMs and demonstrate that DCGen achieves up to a 14%
improvement in visual similarity over competing methods. To the best of our
knowledge, DCGen is the first segment-aware prompt-based approach for
generating UI code directly from screenshots.

该论文提出了 DCGen，一种基于分而治之的方法，用于自动将网页设计转化为 UI 代码。在真实世界的网站数据集和各种多模态大语言模型的广泛测试中，DCGen 在视觉相似性方面的性能较竞争方法提高了 14%。据我们所知，DCGen 是第一种从屏幕截图直接生成 UI 代码的分段感知提示方法。

自自动生成界面代码：基于分而治之方法

Automatically Generating UI Code from Screenshot: A  Divide-and-Conquer-Based Approach

With generative AI advances, the exciting potential for autonomous agents to
manage daily tasks via natural language commands has emerged. However, cur rent
agents are primarily created and tested in simplified synthetic environments,
substantially limiting real-world scenario representation. In this paper, we
build an environment for agent command and control that is highly realistic and
reproducible. Specifically, we focus on agents that perform tasks on websites,
and we create an environment with fully functional websites from four common
domains: e-commerce, social forum discussions, collaborative software
development, and content management. Our environment is enriched with tools
(e.g., a map) and external knowledge bases (e.g., user manuals) to encourage
human-like task-solving. Building upon our environment, we release a set of
benchmark tasks focusing on evaluating the functional correctness of task
completions. The tasks in our benchmark are diverse, long-horizon, and are
designed to emulate tasks that humans routinely perform on the internet. We
design and implement several autonomous agents, integrating recent techniques
such as reasoning before acting. The results demonstrate that solving complex
tasks is challenging: our best GPT-4-based agent only achieves an end-to-end
task success rate of 10.59%. These results highlight the need for further
development of robust agents, that current state-of-the-art LMs are far from
perfect performance in these real-life tasks, and that WebArena can be used to
measure such progress. Our code, data, environment reproduction resources, and
video demonstrations are publicly available at this https URL

建立一个高度逼真和可重现的环境，专注于在网站上执行任务的智能代理，提供一组多样化、长期规划、模拟人类在互联网上例行执行的任务的基准任务以评估任务完成的功能正确性。