Some prior work has shown that LLMs perform well in NLG evaluation for
different tasks. However, we discover that LLMs seem to confuse different
evaluation criteria, which reduces their reliability. For further verification,
we first consider avoiding issues of inconsistent conceptualization and vague
expression in existing NLG quality criteria themselves. So we summarize a clear
hierarchical classification system for 11 common aspects with corresponding
different criteria from previous studies involved. Inspired by behavioral
testing, we elaborately design 18 types of aspect-targeted perturbation attacks
for fine-grained analysis of the evaluation behaviors of different LLMs. We
also conduct human annotations beyond the guidance of the classification system
to validate the impact of the perturbations. Our experimental results reveal
confusion issues inherent in LLMs, as well as other noteworthy phenomena, and
necessitate further research and improvements for LLM-based evaluation.

通过设计和实际测试，我们发现 LLMs 在自然语言生成（NLG）评估中存在混淆不同评估标准的问题，这降低了它们的可靠性。为了进一步验证，我们首先总结了一个清晰的层次分类系统，包括 11 个常见方面的相关标准，然后设计了 18 种针对不同 LLMs 评估行为的攻击方法，并进行人工标注验证，揭示了 LLMs 固有的混淆问题以及其他值得关注的现象，从而为基于 LLMs 的评估提出了进一步的研究和改进的必要性。

基于 LLM 的评估者是否混淆了 NLG 质量标准？

Are LLM-based Evaluators Confusing NLG Quality Criteria?

The scholarly publication space is growing steadily not just in numbers but
also in complexity due to collaboration between individuals from within and
across fields of research. This paper presents a hierarchical classification
system that automatically categorizes a scholarly publication using its
abstract into a three-tier hierarchical label set of fields
(discipline-field-subfield). This system enables a holistic view about the
interdependence of research activities in the mentioned hierarchical tiers in
terms of knowledge production through articles and impact through citations.
The classification system (44 disciplines - 738 fields - 1,501 subfields)
utilizes and is able to cope with 160 million abstract snippets in Microsoft
Academic Graph (Version 2018-05-17) using batch training in a modularized and
distributed fashion to address and assess interdisciplinarity and inter-field
classifications. In addition, we have explored multi-class classifications in
both the single-label and multi-label settings. In total, we have conducted
3,140 experiments, in all models (Convolutional Neural Networks, Recurrent
Neural Networks, Transformers), the classification accuracy is > 90% in 77.84%
and 78.83% of the single-label and multi-label classifications, respectively.
We examine the advantages of our classification by its ability to better align
research texts and output with disciplines, to adequately classify them in an
automated way, as well as to capture the degree of interdisciplinarity in a
publication which enables downstream analytics such as field
interdisciplinarity. This system (a set of pretrained models) can serve as a
backbone to an interactive system of indexing scientific publications.

通过利用抽象、将学术出版物自动分类为专业领域体系结构的一种基于模块化和分布式批量训练的层次分类系统 <discipline-field-subfield>，从而在知识生产和引用间捕捉相互依存的关系，进而更好地对学术出版物进行索引。