Recently, the community has witnessed the advancement of Large Language
Models (LLMs), which have shown remarkable performance on various downstream
tasks. Led by powerful models like ChatGPT and Claude, LLMs are revolutionizing
how users engage with software, assuming more than mere tools but intelligent
assistants. Consequently, evaluating LLMs' anthropomorphic capabilities becomes
increasingly important in contemporary discourse. Utilizing the emotion
appraisal theory from psychology, we propose to evaluate the empathy ability of
LLMs, i.e., how their feelings change when presented with specific situations.
After a careful and comprehensive survey, we collect a dataset containing over
400 situations that have proven effective in eliciting the eight emotions
central to our study. Categorizing the situations into 36 factors, we conduct a
human evaluation involving more than 1,200 subjects worldwide. With the human
evaluation results as references, our evaluation includes five LLMs, covering
both commercial and open-source models, including variations in model sizes,
featuring the latest iterations, such as GPT-4 and LLaMA 2. A conclusion can be
drawn from the results that, despite several misalignments, LLMs can generally
respond appropriately to certain situations. Nevertheless, they fall short in
alignment with the emotional behaviors of human beings and cannot establish
connections between similar situations. Our collected dataset of situations,
the human evaluation results, and the code of our testing framework, dubbed
EmotionBench, is made publicly in this https URL
We aspire to contribute to the advancement of LLMs regarding better alignment
with the emotional behaviors of human beings, thereby enhancing their utility
and applicability as intelligent assistants.

利用心理学中的情感评估理论，我们提出了一种评估大型语言模型（LLMs）的同理心能力的方法，通过在具体情境中观察其感受变化。我们收集了超过 400 个情境，根据 8 种情绪将其分成 36 个因素，并进行了包含超过 1200 名全球参与者的人工评价实验。研究结果显示，尽管存在一些偏差，LLMs 总体能够适当地对特定情况做出反应，但仍然无法与人类的情绪行为建立联系。我们公开了情境数据集、人工评估结果以及我们的测试框架 EmotionBench 的代码，旨在提高 LLMs 在与人类情绪行为的对齐方面的应用价值和实用性。