This paper revisits the problem of complex word identification (CWI)
following up the SemEval CWI shared task. We use ensemble classifiers to
investigate how well computational methods can discriminate between complex and
non-complex words. Furthermore, we analyze the classification performance to
understand what makes lexical complexity challenging. Our findings show that
most systems performed poorly on the SemEval CWI dataset, and one of the
reasons for that is the way in which human annotation was performed.

本研究重新审视复杂词汇鉴别问题，使用集成分类器研究计算方法能否有效区分复杂和非复杂词汇，并分析分类性能以了解词汇复杂性的挑战性原因，发现大多数系统在 SemEval CWI 数据集上表现不佳，其中一个原因是人类标注方式。

复杂词汇识别：数据注释和系统性能挑战

Complex Word Identification: Challenges in Data Annotation and System  Performance

Open-domain human-computer conversation has been attracting increasing
attention over the past few years. However, there does not exist a standard
automatic evaluation metric for open-domain dialog systems; researchers usually
resort to human annotation for model evaluation, which is time- and
labor-intensive. In this paper, we propose RUBER, a Referenced metric and
Unreferenced metric Blended Evaluation Routine, which evaluates a reply by
taking into consideration both a groundtruth reply and a query (previous
user-issued utterance). Our metric is learnable, but its training does not
require labels of human satisfaction. Hence, RUBER is flexible and extensible
to different datasets and languages. Experiments on both retrieval and
generative dialog systems show that RUBER has a high correlation with human
annotation.

本文提出 RUBER 作为一个自动评估指标，旨在为开放式对话系统提供标准的评估方法，其评估响应时综合考虑了真实响应和查询的相关性，无需人工标注，适用于不同的数据集和语言，并通过实验与人工标注结果具有很高的相关性。