Recent advancements in generative Large Language Models(LLMs) have been
remarkable, however, the quality of the text generated by these models often
reveals persistent issues. Evaluating the quality of text generated by these
models, especially in open-ended text, has consistently presented a significant
challenge. Addressing this, recent work has explored the possibility of using
LLMs as evaluators. While using a single LLM as an evaluation agent shows
potential, it is filled with significant uncertainty and instability. To
address these issues, we propose the MATEval: A "Multi-Agent Text Evaluation
framework" where all agents are played by LLMs like GPT-4. The MATEval
framework emulates human collaborative discussion methods, integrating multiple
agents' interactions to evaluate open-ended text. Our framework incorporates
self-reflection and Chain-of-Thought (CoT) strategies, along with feedback
mechanisms, enhancing the depth and breadth of the evaluation process and
guiding discussions towards consensus, while the framework generates
comprehensive evaluation reports, including error localization, error types and
scoring. Experimental results show that our framework outperforms existing
open-ended text evaluation methods and achieves the highest correlation with
human evaluation, which confirms the effectiveness and advancement of our
framework in addressing the uncertainties and instabilities in evaluating
LLMs-generated text. Furthermore, our framework significantly improves the
efficiency of text evaluation and model iteration in industrial scenarios.

通过 MATEval 框架使用生成型大型语言模型进行多智能体文本评估，针对评估开放性文本中的不确定性和不稳定性问题进行改进，并取得了与人类评估最高的相关性，大幅提高了工业场景中的文本评估和模型迭代效率。