Automatic evaluation metrics have been facilitating the rapid development of
automatic summarization methods by providing instant and fair assessments of
the quality of summaries. Most metrics have been developed for the general
domain, especially news and meeting notes, or other language-generation tasks.
However, these metrics are applied to evaluate summarization systems in
different domains, such as biomedical question summarization. To better
understand whether commonly used evaluation metrics are capable of evaluating
automatic summarization in the biomedical domain, we conduct human evaluations
of summarization quality from four different aspects of a biomedical question
summarization task. Based on human judgments, we identify different noteworthy
features for current automatic metrics and summarization systems as well. We
also release a dataset of our human annotations to aid the research of
summarization evaluation metrics in the biomedical domain.

本研究针对生物医学领域的问答摘要任务，从四个不同角度进行了人工评估，用以评估现有的自动化摘要评估指标和系统的可行性，并公布了人工注释数据集，以促进生物医学领域的摘要评估方法的研究。

重新审视生物医学领域中自动问题摘要评估

Revisiting Automatic Question Summarization Evaluation in the Biomedical Domain

We present SacreROUGE, an open-source library for using and developing
summarization evaluation metrics. SacreROUGE removes many obstacles that
researchers face when using or developing metrics: (1) The library provides
Python wrappers around the official implementations of existing evaluation
metrics so they share a common, easy-to-use interface; (2) it provides
functionality to evaluate how well any metric implemented in the library
correlates to human-annotated judgments, so no additional code needs to be
written for a new evaluation metric; and (3) it includes scripts for loading
datasets that contain human judgments so they can easily be used for
evaluation. This work describes the design of the library, including the core
Metric interface, the command-line API for evaluating summarization models and
metrics, and the scripts to load and reformat publicly available datasets. The
development of SacreROUGE is ongoing and open to contributions from the
community.

SacreROUGE 是一个开源库，提供 Python 封装的现有摘要评估指标实现的共同易用接口以及对人工注释判定的相关性评估功能，还包括可加载数据集的脚本，以用于简化评价。

SacreROUGE: 一个开源库，用于使用和开发摘要评估指标

SacreROUGE: An Open-Source Library for Using and Developing  Summarization Evaluation Metrics

Existing Natural Language Generation (NLG) systems are weak AI systems and
exhibit limited capabilities when language generation tasks demand higher
levels of creativity, originality and brevity. Effective solutions or, at least
evaluations of modern NLG paradigms for such creative tasks have been elusive,
unfortunately. This paper introduces and addresses the task of coherent story
generation from independent descriptions, describing a scene or an event.
Towards this, we explore along two popular text-generation paradigms -- (1)
Statistical Machine Translation (SMT), posing story generation as a translation
problem and (2) Deep Learning, posing story generation as a sequence to
sequence learning problem. In SMT, we chose two popular methods such as phrase
based SMT (PB-SMT) and syntax based SMT (SYNTAX-SMT) to `translate' the
incoherent input text into stories. We then implement a deep recurrent neural
network (RNN) architecture that encodes sequence of variable length input
descriptions to corresponding latent representations and decodes them to
produce well formed comprehensive story like summaries. The efficacy of the
suggested approaches is demonstrated on a publicly available dataset with the
help of popular machine translation and summarization evaluation metrics.

本论文提出并解决了通过两种文本生成范例 —— 统计机器翻译和深度学习，生成连贯故事的任务。使用两种 SMT 方法和一个深度 RNN 架构，将可变长度的输入描述编码为对应的潜在表示，并解码以生成形式完备的故事总结。在一个公开数据集上，利用常见的机器翻译和总结评估方法证明了所建议方法的有效性。