Automatic evaluation metrics have been facilitating the rapid development of
automatic summarization methods by providing instant and fair assessments of
the quality of summaries. Most metrics have been developed for the general
domain, especially news and meeting notes, or other language-generation tasks.
However, these metrics are applied to evaluate summarization systems in
different domains, such as biomedical question summarization. To better
understand whether commonly used evaluation metrics are capable of evaluating
automatic summarization in the biomedical domain, we conduct human evaluations
of summarization quality from four different aspects of a biomedical question
summarization task. Based on human judgments, we identify different noteworthy
features for current automatic metrics and summarization systems as well. We
also release a dataset of our human annotations to aid the research of
summarization evaluation metrics in the biomedical domain.

本研究针对生物医学领域的问答摘要任务，从四个不同角度进行了人工评估，用以评估现有的自动化摘要评估指标和系统的可行性，并公布了人工注释数据集，以促进生物医学领域的摘要评估方法的研究。