In this paper, we explore the impact of augmenting pre-trained
Encoder-Decoder models, specifically T5, with linguistic knowledge for the
prediction of a target task. In particular, we investigate whether fine-tuning
a T5 model on an intermediate task that predicts structural linguistic
properties of sentences modifies its performance in the target task of
predicting sentence-level complexity. Our study encompasses diverse experiments
conducted on Italian and English datasets, employing both monolingual and
multilingual T5 models at various sizes. Results obtained for both languages
and in cross-lingual configurations show that linguistically motivated
intermediate fine-tuning has generally a positive impact on target task
performance, especially when applied to smaller models and in scenarios with
limited data availability.

本文探讨了将预训练的编码 - 解码模型（特别是 T5）与语言知识相结合用于预测目标任务时的影响。我们研究了在预测句子的结构语言属性的中间任务上微调 T5 模型是否会改变其在预测句子级复杂性目标任务上的性能。我们的研究包括在意大利语和英语数据集上进行的各种实验，使用了单语和多语 T5 模型以及不同尺寸的模型。无论是在两种语言还是在跨语言配置中，结果表明，从语言学角度出发的中间微调通常对目标任务的性能有积极影响，特别是在模型较小且数据有限的情况下。

语言知识可以增强编码器 - 解码器模型（如果你愿意）

Linguistic Knowledge Can Enhance Encoder-Decoder Models (If You Let It)

With the explosive growth of textual information, summarization systems have
become increasingly important. This work aims at indicating the current state
of the art in abstractive text summarization concisely. As part of this, we
outline the current paradigm shifts towards pre-trained encoder-decoder models
and large autoregressive language models. Additionally, we delve further into
the challenges of evaluating summarization systems and the potential of
instruction-tuned models for zero-shot summarization. Finally, we provide a
brief overview of how summarization systems are currently being integrated into
commercial applications.

本文旨在简明扼要地阐述抽象文本摘要现状，强调需采用预训练编码器 - 解码器模型与大型自回归语言模型。此外，讨论了摘要系统评估的挑战和用于零 - shot 摘要的 instruction tuned 模型的潜力，最后概述了如何将摘要系统整合到商业应用中。

摘要现状

The Current State of Summarization

This study presents three deidentified large medical text datasets, named
DISCHARGE, ECHO and RADIOLOGY, which contain 50K, 16K and 378K pairs of report
and summary that are derived from MIMIC-III, respectively. We implement
convincing baselines of automated abstractive summarization on the proposed
datasets with pre-trained encoder-decoder language models, including BERT2BERT,
T5-large and BART. Further, based on the BART model, we leverage the sampled
summaries from the train set as prior knowledge guidance, for encoding
additional contextual representations of the guidance with the encoder and
enhancing the decoding representations in the decoder. The experimental results
confirm the improvement of ROUGE scores and BERTScore made by the proposed
method, outperforming the larger model T5-large.

本研究提出了三个去识别化的医学文本数据集，命名为 DISCHARGE、ECHO 和 RADIOLOGY，共包含 50K、16K 和 378K 的报告和摘要对，作者在这些数据集上使用基于预训练编码器 - 解码器语言模型的有说服力的自动抽象摘要基线，并基于 BART 模型利用训练集的示例摘要作为先验知识引导来增强编码器和解码器的上下文表示，实验结果证实了这种方法在 ROUGE 分数和 BERTScore 上的改进，超过了更大模型 T5-large。