Pretrained transformer-based models have shown high performance in natural
language generation task. However, a new wave of interest has surged: automatic
programming language generation. This task consists of translating natural
language instructions to a programming code. Despite the fact that well-known
pretrained models on language generation have achieved good performance in
learning programming languages, effort is still needed in automatic code
generation. In this paper, we introduce JaCoText, a model based on Transformers
neural network. It aims to generate java source code from natural language
text. JaCoText leverages advantages of both natural language and code
generation models. More specifically, we study some findings from the state of
the art and use them to (1) initialize our model from powerful pretrained
models, (2) explore additional pretraining on our java dataset, (3) carry out
experiments combining the unimodal and bimodal data in the training, and (4)
scale the input and output length during the fine-tuning of the model.
Conducted experiments on CONCODE dataset show that JaCoText achieves new
state-of-the-art results.

本文介绍了一种基于 Transformer 神经网络的模型 JaCoText，旨在将自然语言文本生成 java 源代码。通过在强大的预训练模型上初始化，探索我们的 java 数据集上的额外预训练，以及在训练中结合单模态和双模态数据的实验，以及在模型的微调期间缩放输入和输出长度等方法，经过 CONCODE 数据集的实验表明，JaCoText 取得了新的最先进的结果。