Automated HTML/CSS code generation from screenshots is an important yet
challenging problem with broad applications in website development and design.
In this paper, we present a novel vision-code transformer approach that
leverages an Encoder-Decoder architecture as well as explore actor-critic
fine-tuning as a method for improving upon the baseline. For this purpose, two
image encoders are compared: Vision Transformer (ViT) and Document Image
Transformer (DiT).
We propose an end-to-end pipeline that can generate high-quality code
snippets directly from screenshots, streamlining the website creation process
for developers. To train and evaluate our models, we created a synthetic
dataset of 30,000 unique pairs of code and corresponding screenshots.
We evaluate the performance of our approach using a combination of automated
metrics such as MSE, BLEU, IoU, and a novel htmlBLEU score, where our models
demonstrated strong performance. We establish a strong baseline with the
DiT-GPT2 model and show that actor-critic can be used to improve IoU score from
the baseline of 0.64 to 0.79 and lower MSE from 12.25 to 9.02. We achieved
similar performance as when using larger models, with much lower computational
cost.

本文提出了一种新颖的视觉 - 代码转换器方法，并探索了 Actor-Critic 微调作为提高基线的一种方法，用于从屏幕截图生成高质量代码片段，其性能表现突出，在自动化度量方面表现强大。

强化学习微调的视觉 - 编码转换器用于 UI 转码生成

Reinforcement Learning finetuned Vision-Code Transformer for UI-to-Code  Generation

Image Transformer has recently achieved significant progress for natural
image understanding, either using supervised (ViT, DeiT, etc.) or
self-supervised (BEiT, MAE, etc.) pre-training techniques. In this paper, we
propose \textbf{DiT}, a self-supervised pre-trained \textbf{D}ocument
\textbf{I}mage \textbf{T}ransformer model using large-scale unlabeled text
images for Document AI tasks, which is essential since no supervised
counterparts ever exist due to the lack of human-labeled document images. We
leverage DiT as the backbone network in a variety of vision-based Document AI
tasks, including document image classification, document layout analysis, table
detection as well as text detection for OCR. Experiment results have
illustrated that the self-supervised pre-trained DiT model achieves new
state-of-the-art results on these downstream tasks, e.g. document image
classification (91.11 $\rightarrow$ 92.69), document layout analysis (91.0
$\rightarrow$ 94.9), table detection (94.23 $\rightarrow$ 96.55) and text
detection for OCR (93.07 $\rightarrow$ 94.29). The code and pre-trained models
are publicly available at https://aka.ms/msdit.

本文提出了 DiT，一种利用大规模未标记文本图像进行自监督预训练的文档图像变压器模型，成为视觉文档 AI 任务的骨干网络，在文档图像分类、文档布局分析、表格检测以及 OCR 的文本检测等诸多任务中取得了最新的最佳结果。

DiT：文档图像 Transformer 的自监督预训练

DiT: Self-supervised Pre-training for Document Image Transformer

In this work, we propose a new framework, called Document Image Transformer
(DocTr), to address the issue of geometry and illumination distortion of the
document images. Specifically, DocTr consists of a geometric unwarping
transformer and an illumination correction transformer. By setting a set of
learned query embedding, the geometric unwarping transformer captures the
global context of the document image by self-attention mechanism and decodes
the pixel-wise displacement solution to correct the geometric distortion. After
geometric unwarping, our illumination correction transformer further removes
the shading artifacts to improve the visual quality and OCR accuracy. Extensive
evaluations are conducted on several datasets, and superior results are
reported against the state-of-the-art methods. Remarkably, our DocTr achieves
20.02% Character Error Rate (CER), a 15% absolute improvement over the
state-of-the-art methods. Moreover, it also shows high efficiency on running
time and parameter count. The results will be available at
this https URL for further comparison.

本文提出了一种新的框架，称为文档图像变压器 (DocTr)，以解决文档图像的几何和光照失真问题。 具体来说，DocTr 由几何展开变压器和光照校正变压器组成。 通过设置一组学习的查询嵌入，几何展开变压器通过自注意机制捕获文档图像的全局上下文，并解码逐像素的位移解以校正几何失真。在几何展开后，我们的光照校正变压器进一步去除阴影伪影，以改善视觉质量和 OCR 准确性。我们在几个数据集上进行了广泛的评估，并报告了针对最先进方法的优越结果。值得注意的是，我们的 DocTr 实现了 20.02％的字符错误率 (CER)，比最先进的方法提高了 15％绝对值。 此外，它还显示出运行时间和参数计数的高效性。