Sketches are a natural and accessible medium for UI designers to conceptualize early-stage ideas. However, existing research on UI/UX automation often requires high-fidelity inputs like Figma designs or detailed screenshots, limiting accessibility and impeding efficient design iteration. To bridge this gap, we introduce Sketch2Code, a benchmark that evaluates state-of-the-art Vision Language Models (VLMs) on automating the conversion of rudimentary sketches into webpage prototypes. Beyond end-to-end benchmarking, Sketch2Code supports interactive agent evaluation that mimics real-world design workflows, where a VLM-based agent iteratively refines its generations by communicating with a simulated user, either passively receiving feedback instructions or proactively asking clarification questions. We comprehensively analyze ten commercial and open-source models, showing that Sketch2Code is challenging for existing VLMs; even the most capable models struggle to accurately interpret sketches and formulate effective questions that lead to steady improvement. Nevertheless, a user study with UI/UX experts reveals a significant preference for proactive question-asking over passive feedback reception, highlighting the need to develop more effective paradigms for multi-turn conversational agents.

本研究针对现有UI/UX自动化研究中对高保真输入的依赖问题，提出Sketch2Code基准，评估先进的视觉语言模型将简单草图转换为网页原型的能力。研究发现，现有模型在准确解读草图和提出有效问题方面存在挑战，而用户研究显示，主动提问比被动反馈更受UI/UX专家青睐，这表明需发展更有效的多轮对话代理模型。

Sketch2Code：评估视觉语言模型在互动网页设计原型制作中的应用