Human evaluation plays a crucial role in Natural Language Processing (NLP) as
it assesses the quality and relevance of developed systems, thereby
facilitating their enhancement. However, the absence of widely accepted human
evaluation metrics in NLP hampers fair comparisons among different systems and
the establishment of universal assessment standards. Through an extensive
analysis of existing literature on human evaluation metrics, we identified
several gaps in NLP evaluation methodologies. These gaps served as motivation
for developing our own hierarchical evaluation framework. The proposed
framework offers notable advantages, particularly in providing a more
comprehensive representation of the NLP system's performance. We applied this
framework to evaluate the developed Machine Reading Comprehension system, which
was utilized within a human-AI symbiosis model. The results highlighted the
associations between the quality of inputs and outputs, underscoring the
necessity to evaluate both components rather than solely focusing on outputs.
In future work, we will investigate the potential time-saving benefits of our
proposed framework for evaluators assessing NLP systems.

通过对现有文献进行广泛分析，我们发现在自然语言处理的人工评估方法中存在一些缺陷，这促使我们开发了一种层次化评估框架，该框架具有更全面地表示自然语言处理系统性能的显著优点，并应用于评估机器阅读理解系统及其与人工智能共生模型中的输入和输出的质量之间的关联。