Academic literature does not give much guidance on how to build the best possible customer-facing summarization system from existing research components. Here we present analyses to inform the selection of a system backbone from popular models; we find that in both automatic and human evaluation, BART performs better than PEGASUS and T5. We also find that when applied cross-domain, summarizers exhibit considerably worse performance. At the same time, a system fine-tuned on heterogeneous domains performs well on all domains and will be most suitable for a broad-domain summarizer. Our work highlights the need for heterogeneous domain summarization benchmarks. We find considerable variation in system output that can be captured only with human evaluation and are thus unlikely to be reflected in standard leaderboards with only automatic evaluation.

本研究分析了现有研究组件以选择系统骨干，发现BART比PEGASUS和T5表现更好；跨域总结器的表现要差得多，而在异构领域进行微调的系统在所有领域上表现良好。该研究强调了对异构领域摘要基准的重要性。系统输出存在相当大的差异，只有人工评估才能捕捉到，因此不太可能反映在只有自动评估的标准排行榜中。

从排行榜到实践的摘要生成：选择一种表示方法的骨干和确保稳健性