Can pre-trained BERT for one language and GPT for another be glued together
to translate texts? Self-supervised training using only monolingual data has
led to the success of pre-trained (masked) language models in many NLP tasks.
However, directly connecting BERT as an encoder and GPT as a decoder can be
challenging in machine translation, for GPT-like models lack a cross-attention
component that is needed in seq2seq decoders. In this paper, we propose
Graformer to graft separately pre-trained (masked) language models for machine
translation. With monolingual data for pre-training and parallel data for
grafting training, we maximally take advantage of the usage of both types of
data. Experiments on 60 directions show that our method achieves average
improvements of 5.8 BLEU in x2en and 2.9 BLEU in en2x directions comparing with
the multilingual Transformer of the same size.

本文提出了 Graformer 模型，将预训练的 BERT 和 GPT 结合起来，分别作为编码器和解码器用于机器翻译，得到了比同等规模的多语言 Transformer 更好的翻译结果。

利用预训练语言模型嫁接的多语言翻译

Multilingual Translation via Grafting Pre-trained Language Models

We present Graformer, a novel Transformer-based encoder-decoder architecture
for graph-to-text generation. With our novel graph self-attention, the encoding
of a node relies on all nodes in the input graph - not only direct neighbors -
facilitating the detection of global patterns. We represent the relation
between two nodes as the length of the shortest path between them. Graformer
learns to weight these node-node relations differently for different attention
heads, thus virtually learning differently connected views of the input graph.
We evaluate Graformer on two popular graph-to-text generation benchmarks,
AGENDA and WebNLG, where it achieves strong performance while using many fewer
parameters than other approaches.

本文介绍了 Graformer，一种新颖的基于 Transformer 的编码器 - 解码器体系结构，用于图形到文本的生成。使用我们新颖的图形自我关注，节点的编码依赖于输入图中的所有节点，促进了全局模式的检测。通过学习不同关注头的节点之间的权重，Graformer 实现了不同的连接视图，从而在 AGENDA 和 WebNLG 这两个流行的图形到文本生成基准测试中实现强大的性能。