Automatic Program translation has enormous application value and hence has
been attracting significant interest from AI researchers. However, we observe
that current program translation models still make elementary syntax errors,
particularly, when the target language does not have syntax elements in the
source language. Metrics like BLUE, CodeBLUE and computation accuracy may not
expose these issues. In this paper we introduce a new metrics for programming
language translation and these metrics address these basic syntax errors. We
develop a novel active defects probing suite called Syntactic Unit Tests (SUT)
which includes a highly interpretable evaluation harness for accuracy and test
scoring. Experiments have shown that even powerful models like ChatGPT still
make mistakes on these basic unit tests. Specifically, compared to previous
program translation task evaluation dataset, its pass rate on our unit tests
has decreased by 26.15%. Further our evaluation harness reveal syntactic
element errors in which these models exhibit deficiencies.

自动程序翻译具有巨大的应用价值，但当前的翻译模型在基本的语法错误方面仍存在问题。本论文介绍了针对编程语言翻译的新指标和一套包括高度可解释的评估工具的测试套件，实验证明即使是像 ChatGPT 这样强大的模型在基本单元测试上也存在错误。

SUT: 转换器模型的主动缺陷探测

SUT: Active Defects Probing for Transcompiler Models

There is growing interest in software migration as the development of
software and society. Manually migrating projects between languages is
error-prone and expensive. In recent years, researchers have begun to explore
automatic program translation using supervised deep learning techniques by
learning from large-scale parallel code corpus. However, parallel resources are
scarce in the programming language domain, and it is costly to collect
bilingual data manually. To address this issue, several unsupervised
programming translation systems are proposed. However, these systems still rely
on huge monolingual source code to train, which is very expensive. Besides,
these models cannot perform well for translating the languages that are not
seen during the pre-training procedure. In this paper, we propose SDA-Trans, a
syntax and domain-aware model for program translation, which leverages the
syntax structure and domain knowledge to enhance the cross-lingual transfer
ability. SDA-Trans adopts unsupervised training on a smaller-scale corpus,
including Python and Java monolingual programs. The experimental results on
function translation tasks between Python, Java, and C++ show that SDA-Trans
outperforms many large-scale pre-trained models, especially for unseen language
translation.

本文提出了一种语法和领域感知的程序翻译模型 SDA-Trans，利用语法结构和领域知识来增强跨语言转移能力，采用较小规模的无监督语料库进行训练，实验结果表明 SDA-Trans 在 Python、Java 和 C++ 之间的函数翻译任务上表现优异，尤其是对于需要翻译看不见语言的情况。