This review gives an extensive overview of evaluation methods for
task-oriented dialogue systems, paying special attention to practical
applications of dialogue systems, for example for customer service. The review
(1) provides an overview of the used constructs and metrics in previous work,
(2) discusses challenges in the context of dialogue system evaluation and (3)
develops a research agenda for the future of dialogue system evaluation. We
conducted a systematic review of four databases (ACL, ACM, IEEE and Web of
Science), which after screening resulted in 122 studies. Those studies were
carefully analysed for the constructs and methods they proposed for evaluation.
We found a wide variety in both constructs and methods. Especially the
operationalisation is not always clearly reported. We hope that future work
will take a more critical approach to the operationalisation and specification
of the used constructs. To work towards this aim, this review ends with
recommendations for evaluation and suggestions for outstanding questions.

对任务导向型对话系统的评估方法进行广泛的综述，着重关注对话系统在实际应用中（例如客户服务）的应用；对以前工作中使用的构造和指标进行了概述；讨论了对话系统评估的挑战，并制定了未来研究议程；通过对四个数据库（ACL、ACM、IEEE 和 Web of Science）的系统综述，总共有 122 项研究进行了仔细分析，发现没有明确报告操作化情况，并希望未来的研究能对使用的构造进行更加批判性的操作化和规范化；最后给出了评估方面的建议和未解决问题的建议。