Federated Learning (FL) has emerged as a promising solution for collaborative
training of large language models (LLMs). However, the integration of LLMs into
FL introduces new challenges, particularly concerning the evaluation of LLMs.
Traditional evaluation methods that rely on labeled test sets and
similarity-based metrics cover only a subset of the acceptable answers, thereby
failing to accurately reflect the performance of LLMs on generative tasks.
Meanwhile, although automatic evaluation methods that leverage advanced LLMs
present potential, they face critical risks of data leakage due to the need to
transmit data to external servers and suboptimal performance on downstream
tasks due to the lack of domain knowledge. To address these issues, we propose
a Federated Evaluation framework of Large Language Models, named FedEval-LLM,
that provides reliable performance measurements of LLMs on downstream tasks
without the reliance on labeled test sets and external tools, thus ensuring
strong privacy-preserving capability. FedEval-LLM leverages a consortium of
personalized LLMs from participants as referees to provide domain knowledge and
collective evaluation capability, thus aligning to the respective downstream
tasks and mitigating uncertainties and biases associated with a single referee.
Experimental results demonstrate a significant improvement in the evaluation
capability of personalized evaluation models on downstream tasks. When applied
to FL, these evaluation models exhibit strong agreement with human preference
and RougeL-score on meticulously curated test sets. FedEval-LLM effectively
overcomes the limitations of traditional metrics and the reliance on external
services, making it a promising framework for the evaluation of LLMs within
collaborative training scenarios.

提出了一种名为 FedEval-LLM 的大型语言模型联邦评估框架，通过利用参与者提供的个性化语言模型，提供了一种可靠的大型语言模型在下游任务中的性能测量方式，确保了强大的隐私保护能力，解决了传统评估方法和对外部服务的依赖性的限制，是协作训练场景中语言模型评估的一种有前景的框架。