When translating UI design prototypes to code in industry, automatically
generating code from design prototypes can expedite the development of
applications and GUI iterations. However, in design prototypes without strict
design specifications, UI components may be composed of fragmented elements.
Grouping these fragmented elements can greatly improve the readability and
maintainability of the generated code. Current methods employ a two-stage
strategy that introduces hand-crafted rules to group fragmented elements.
Unfortunately, the performance of these methods is not satisfying due to
visually overlapped and tiny UI elements. In this study, we propose EGFE, a
novel method for automatically End-to-end Grouping Fragmented Elements via UI
sequence prediction. To facilitate the UI understanding, we innovatively
construct a Transformer encoder to model the relationship between the UI
elements with multi-modal representation learning. The evaluation on a dataset
of 4606 UI prototypes collected from professional UI designers shows that our
method outperforms the state-of-the-art baselines in the precision (by
29.75\%), recall (by 31.07\%), and F1-score (by 30.39\%) at edit distance
threshold of 4. In addition, we conduct an empirical study to assess the
improvement of the generated front-end code. The results demonstrate the
effectiveness of our method on a real software engineering application. Our
end-to-end fragmented elements grouping method creates opportunities for
improving UI-related software engineering tasks.

通过 UI 序列预测的方式，我们提出了一种自动从设计原型中将碎片化元素进行端到端分组的新方法，通过创新性地构建 Transformer 编码器来改善 UI 理解，我们的方法在 4606 个专业 UI 设计师收集的 UI 原型数据集上优于现有基准，在 4 个编辑距离阈值下，精确度、召回率和 F1 分数分别提高了 29.75%、31.07% 和 30.39%，此外，我们进行了一项实证研究以评估生成的前端代码的改进，结果表明我们的方法在真实软件工程应用中具有显著的效果，我们的端到端碎片化元素分组方法为改进与 UI 相关的软件工程任务提供了机会。