With the emergence of large language models (LLMs), investigating if they can
surpass humans in areas such as emotion recognition and empathetic responding
has become a focal point of research. This paper presents a comprehensive study
exploring the empathetic responding capabilities of four state-of-the-art LLMs:
GPT-4, LLaMA-2-70B-Chat, Gemini-1.0-Pro, and Mixtral-8x7B-Instruct in
comparison to a human baseline. We engaged 1,000 participants in a
between-subjects user study, assessing the empathetic quality of responses
generated by humans and the four LLMs to 2,000 emotional dialogue prompts
meticulously selected to cover a broad spectrum of 32 distinct positive and
negative emotions. Our findings reveal a statistically significant superiority
of the empathetic responding capability of LLMs over humans. GPT-4 emerged as
the most empathetic, marking approximately 31% increase in responses rated as
"Good" compared to the human benchmark. It was followed by LLaMA-2,
Mixtral-8x7B, and Gemini-Pro, which showed increases of approximately 24%, 21%,
and 10% in "Good" ratings, respectively. We further analyzed the response
ratings at a finer granularity and discovered that some LLMs are significantly
better at responding to specific emotions compared to others. The suggested
evaluation framework offers a scalable and adaptable approach for assessing the
empathy of new LLMs, avoiding the need to replicate this study's findings in
future research.

本研究通过与人类基准对比的方式，综合评估了四个最先进的大型语言模型（GPT-4、LLaMA-2、Mixtral-8x7B 和 Gemini-Pro）在共 2000 个情感对话提示中的回应中的移情能力。我们的发现显示，大型语言模型在移情回应能力上显著超过人类，其中 GPT-4 的回应被评为 “好” 的比例比人类基准提高了约 31%。此外，我们发现不同的大型语言模型在回应不同情感时表现显著不同。基于研究结果，我们提出了一种可扩展和可适应的评估框架，用于评估新大型语言模型的移情能力，避免了未来研究中重复这项研究的需求。

大型语言模型是否比人类更具移情能力？

Are Large Language Models More Empathetic than Humans?

This paper investigates the empathetic responding capabilities of ChatGPT,
particularly its latest iteration, GPT-4, in comparison to human-generated
responses to a wide range of emotional scenarios, both positive and negative.
We employ a rigorous evaluation methodology, involving a between-groups study
with 600 participants, to evaluate the level of empathy in responses generated
by humans and ChatGPT. ChatGPT is prompted in two distinct ways: a standard
approach and one explicitly detailing empathy's cognitive, affective, and
compassionate counterparts. Our findings indicate that the average empathy
rating of responses generated by ChatGPT exceeds those crafted by humans by
approximately 10%. Additionally, instructing ChatGPT to incorporate a clear
understanding of empathy in its responses makes the responses align
approximately 5 times more closely with the expectations of individuals
possessing a high degree of empathy, compared to human responses. The proposed
evaluation framework serves as a scalable and adaptable framework to assess the
empathetic capabilities of newer and updated versions of large language models,
eliminating the need to replicate the current study's results in future
research.

研究论文通过对 ChatGPT 及其最新版本 GPT-4 与人类生成的情感场景回应能力进行比较，探究了 ChatGPT 在回应不同情绪场景（正面和负面）时的共情水平。研究采用一项涉及 600 名参与者的组间研究来评估 ChatGPT 和人类生成的回应的共情程度。研究发现，ChatGPT 所生成的回应的平均共情评分比人类生成的回应高出约 10％。此外，明确要求 ChatGPT 在回应中融入对共情认知、情感和同情的明确理解，使其回应与具有较高共情度的个体的期望更为接近，相比之下，人类回应的一致性要小 5 倍。该研究提出的评估框架可用于评估较大的语言模型的共情能力，并且具有可扩展性和适应性，避免了未来研究中重复当前研究结果的需要。