There is an increasing trend towards evaluating NLP models with LLM-generated judgments instead of human judgments. In the absence of a comparison against human data, this raises concerns about the validity of these evaluations; in case they are conducted with proprietary models, this also raises concerns over reproducibility. We provide JUDGE-BENCH, a collection of 20 NLP datasets with human annotations, and comprehensively evaluate 11 current LLMs, covering both open-weight and proprietary models, for their ability to replicate the annotations. Our evaluations show that each LLM exhibits a large variance across datasets in its correlation to human judgments. We conclude that LLMs are not yet ready to systematically replace human judges in NLP.

评估NLP模型时，使用LLM-generated判断取而代之人为判断的趋势日益增长。我们提供了一个由人工注释的20个NLP数据集的JUDGE-BENCH，并对11个当前的LLM进行全面评估，涵盖公开权重和专有模型，以验证其模拟注释的能力。我们的评估结果表明，每个LLM在与人工判断的相关性方面在不同数据集上存在很大的差异。我们得出结论，LLM尚不具备系统替代NLP中的人类评审员的能力。

用大型经验研究代替人类法官？跨20个NLP评估任务