It is common practice for developers of user-facing software to transform a
mock-up of a graphical user interface (GUI) into code. This process takes place
both at an application's inception and in an evolutionary context as GUI
changes keep pace with evolving features. Unfortunately, this practice is
challenging and time-consuming. In this paper, we present an approach that
automates this process by enabling accurate prototyping of GUIs via three
tasks: detection, classification, and assembly. First, logical components of a
GUI are detected from a mock-up artifact using either computer vision
techniques or mock-up metadata. Then, software repository mining, automated
dynamic analysis, and deep convolutional neural networks are utilized to
accurately classify GUI-components into domain-specific types (e.g.,
toggle-button). Finally, a data-driven, K-nearest-neighbors algorithm generates
a suitable hierarchical GUI structure from which a prototype application can be
automatically assembled. We implemented this approach for Android in a system
called ReDraw. Our evaluation illustrates that ReDraw achieves an average
GUI-component classification accuracy of 91% and assembles prototype
applications that closely mirror target mock-ups in terms of visual affinity
while exhibiting reasonable code structure. Interviews with industrial
practitioners illustrate ReDraw's potential to improve real development
workflows.

本文介绍一种自动化的 GUI 原型设计方法，通过检测、分类和组装实现准确地从 GUI 设计稿生成代码，该方法利用计算机视觉技术、自动化动态分析和深度卷积神经网络来对组件进行分类，并使用基于数据的 K 最近邻算法生成适当的分层 GUI 结构。在 Android 平台上实现了该方法的系统 ReDraw，并取得了评估成果，ReDraw 实现的平均 GUI 组件分类准确率为 91％，组装的原型应用程序的视觉外观与目标 GUI 设计稿非常相似，同时展现合理的代码结构，多位工业从业人员的访谈表明 ReDraw 能够改善真实的开发流程。