Pretrained language models (PLMs) have produced substantial improvements in
discourse-aware neural machine translation (NMT), for example, improved
coherence in spoken language translation. However, the underlying reasons for
their strong performance have not been well explained. To bridge this gap, we
introduce a probing task to interpret the ability of PLMs to capture discourse
relation knowledge. We validate three state-of-the-art PLMs across encoder-,
decoder-, and encoder-decoder-based models. The analysis shows that (1) the
ability of PLMs on discourse modelling varies from architecture and layer; (2)
discourse elements in a text lead to different learning difficulties for PLMs.
Besides, we investigate the effects of different PLMs on spoken language
translation. Through experiments on IWSLT2017 Chinese-English dataset, we
empirically reveal that NMT models initialized from different layers of PLMs
exhibit the same trends with the probing task. Our findings are instructive to
understand how and when discourse knowledge in PLMs should work for downstream
tasks.

本研究探讨了预训练语言模型在捕捉篇章关系上的能力，并通过探究任务验证了不同架构和层数的 PLMs 的表现，得出了对于不同 NMT 任务在何种情况下使用不同层的 PLMs 是最好的决策。

预训练如何改善基于语篇的翻译？

How Does Pretraining Improve Discourse-Aware Translation?

In recent years, deep natural image matting has been rapidly evolved by
extracting high-level contextual features into the model. However, most current
methods still have difficulties with handling tiny details, like hairs or furs.
In this paper, we argue that recovering these microscopic details relies on
low-level but high-definition texture features. However, {these features are
downsampled in a very early stage in current encoder-decoder-based models,
resulting in the loss of microscopic details}. To address this issue, we design
a deep image matting model {to enhance fine-grained details. Our model consists
of} two parallel paths: a conventional encoder-decoder Semantic Path and an
independent downsampling-free Textural Compensate Path (TCP). The TCP is
proposed to extract fine-grained details such as lines and edges in the
original image size, which greatly enhances the fineness of prediction.
Meanwhile, to leverage the benefits of high-level context, we propose a feature
fusion unit(FFU) to fuse multi-scale features from the semantic path and inject
them into the TCP. In addition, we have observed that poorly annotated trimaps
severely affect the performance of the model. Thus we further propose a novel
term in loss function and a trimap generation method to improve our model's
robustness to the trimaps. The experiments show that our method outperforms
previous start-of-the-art methods on the Composition-1k dataset.

本文提出了一种深度图像抠图模型，其中利用低阶但高清晰度的纹理特征提供微观细节的恢复，并设计了一种独立的不会下采样的 “Textural Compensate Path” 来提取精细的细节，较之前的方法在 “Composition-1k” 数据集上有较好的表现。