Large language models (LLMs) have shown impressive capabilities across various natural language tasks. However, evaluating their alignment with human preferences remains a challenge. To this end, we propose a comprehensive human evaluation framework to assess LLMs' proficiency in following instructions on diverse real-world tasks. We construct a hierarchical task tree encompassing 7 major areas covering over 200 categories and over 800 tasks, which covers diverse capabilities such as question answering, reasoning, multiturn dialogue, and text generation, to evaluate LLMs in a comprehensive and in-depth manner. We also design detailed evaluation standards and processes to facilitate consistent, unbiased judgments from human evaluators. A test set of over 3,000 instances is released, spanning different difficulty levels and knowledge domains. Our work provides a standardized methodology to evaluate human alignment in LLMs for both English and Chinese. We also analyze the feasibility of automating parts of evaluation with a strong LLM (GPT-4). Our framework supports a thorough assessment of LLMs as they are integrated into real-world applications. We have made publicly available the task tree, TencentLLMEval dataset, and evaluation methodology which have been demonstrated as effective in assessing the performance of Tencent Hunyuan LLMs. By doing so, we aim to facilitate the benchmarking of advances in the development of safe and human-aligned LLMs.

通过构建一个综合的人工评估框架，我们提出了一个评估大规模语言模型在不同实际任务中遵循指令的能力的方法，同时设计了详细的评估标准和过程，释放了一个包含不同难度水平和知识领域的测试集，并分析了自动化评估的可行性。我们的研究为评估英语和中文大规模语言模型的人类对齐性提供了一个标准化的方法，旨在促进安全和人类对齐性大规模语言模型发展进步的基准化。

腾讯LLMEval：人类对齐的LLMs的实际能力的层次评估